[Feature request] Add proxy mode to Langfuse for seamless observability

[kha84]

Describe the feature or potential improvement

I would like to propose a new feature for Langfuse – the ability to operate in proxy mode for Ollama-based and other LLM applications. This enhancement would allow Langfuse to intercept and log requests and responses without requiring developers to manually instrument their application code.

By acting as a proxy between the application and Ollama (or other LLM services), Langfuse could provide observability, tracing, and cost tracking out of the box by simply routing traffic through the Langfuse instance.

Additional information

Why This Feature is Valuable:

Currently, integrating Langfuse into LLM applications requires developers to modify their code to wrap API calls or add observability logic. While this approach works well, it introduces some friction, particularly for:

• Applications already in production.
• Teams that want quick observability without altering core business logic.
• Projects with multiple models or endpoints that require consistent logging.

A proxy mode would significantly lower the barrier to adoption by enabling observability with zero code changes to the application itself. Developers would simply update their endpoint to point to Langfuse, which would forward the traffic to the actual Ollama instance while logging all interactions.

Key Benefits for Developers:

1. Zero Code Changes for Observability

   • Applications can be monitored by changing API endpoints (e.g., from localhost:11434 to localhost:8080 for Langfuse proxy), eliminating the need to modify application logic.
   • Ideal for legacy systems or production environments where code changes are not feasible.

2. Unified Observability Without Instrumentation

   • Langfuse would automatically capture prompts, responses, latencies, and errors for all LLM interactions passing through the proxy.
   • Developers can track multiple endpoints and models from a single interface, centralizing observability.

3. Dynamic Request Routing and Load Balancing

   • As a proxy, Langfuse could route requests to different Ollama instances or models, enabling dynamic load balancing or model fallback.
   • This could enhance performance and reliability by distributing traffic intelligently.

4. Security and Access Control

   • Langfuse could implement rate limiting, authentication, and request validation before forwarding to Ollama, adding a layer of protection.
   • Developers could manage API keys and throttle usage directly from Langfuse.

5. Cost Monitoring and Performance Analytics

   • Langfuse could track token usage, latency, and errors in real-time, providing insights into performance and cost-efficiency without modifying application code.
   • This simplifies budgeting and optimization for LLM-heavy applications.

6. Simplified Experimentation (A/B Testing)

   • The proxy could modify or reroute requests dynamically, enabling prompt experimentation or model switching without changing application-side logic.

How This Could Work (Conceptual Flow):

[Client App]  -->  [Langfuse Proxy]  -->  [Ollama API (localhost or cloud)]

• Intercept requests ➝ Log inputs/outputs ➝ Forward to Ollama ➝ Log response ➝ Return to client.
• The proxy could optionally modify requests (e.g., add metadata) or reroute them based on load or custom rules.

Implementation Ideas:

• Configurable Proxy Mode: Enable proxy mode through Langfuse’s config file or CLI.
• Endpoint Mapping: Allow developers to specify the target Ollama endpoint (or other LLM APIs) that Langfuse should forward traffic to.
• TLS Support: Ensure secure traffic forwarding for production deployments.
• Lightweight and Fast: The proxy should introduce minimal latency to avoid impacting LLM performance.

Use Cases:

• Monitoring Ollama Deployments without modifying app logic.
• Rapid Observability for Prototyping and hackathons.
• Production-Grade LLM Applications requiring centralized logging and control.
• Internal Tools that use multiple LLM endpoints, enabling unified observability.

Conclusion:

A proxy mode for Langfuse would provide significant value by simplifying observability and reducing friction for developers integrating LLMs like Ollama. This feature aligns with Langfuse’s mission to enhance LLM observability, making it easier for teams to monitor, debug, and optimize their AI workflows with minimal overhead.

I believe this addition could greatly expand Langfuse’s adoption and usability across a wide range of LLM-powered applications. Thank you for considering this feature request!

[marcklingen]

This is available today by using the excellent LiteLLM OSS proxy which natively integrates with Langfuse:

https://langfuse.com/docs/integrations/litellm/tracing