Open AI compatible servers do not work anymore

[ParisNeo]

Hi there

@danielaskdd, you have made a refactoring of the API which is OK, but that broke some functionalities:

Some people use lightrag with lm studio via OpenAI API.

You have removed the possibility of specifying the host for openai api models resulting in the impossibility to use the lightrag api with an openai compatible service like lm studio.

Would you please do the update so that we can use openai api locally?

Best regards

[danielaskdd]

If someone want to use OpenAI alike LLM, he can setup the server according to lightrag/api/README.md

For OpenAI Alike Server

• Requires environment variables setup or command line argument provided
• Environment variables: LLM_BINDING=ollama, LLM_BINDING_HOST, LLM_MODEL, LLM_BINDING_API_KEY
• Command line arguments: --llm-binding=ollama, --llm-binding-host, --llm-model, --llm-binding-api-key
• Default connection is https://api.openai.com/v1 if not priveded

[danielaskdd]

Here is the .env file example for access Open WebUI's OpenAI compatible API:

# OpenAI alike example
LLM_BINDING=openai
LLM_BINDING_HOST=https://localhost/api
LLM_MODEL=deepseek-chat
LLM_BINDING_API_KEY=your_api_key

[ParisNeo]

Thanks for this. But according to the code, you do not pass the arguments to the LightRAG kwargs in case openai is selected:

    # Initialize RAG
    if args.llm_binding in ["lollms", "ollama"]:
        rag = LightRAG(
            working_dir=args.working_dir,
            llm_model_func=lollms_model_complete
            if args.llm_binding == "lollms"
            else ollama_model_complete,
            llm_model_name=args.llm_model,
            llm_model_max_async=args.max_async,
            llm_model_max_token_size=args.max_tokens,
            llm_model_kwargs={
                "host": args.llm_binding_host,
                "timeout": args.timeout,
                "options": {"num_ctx": args.max_tokens},
                "api_key": args.llm_binding_api_key,
            },
            embedding_func=embedding_func,
        )
    else:
        rag = LightRAG(
            working_dir=args.working_dir,
            llm_model_func=azure_openai_model_complete
            if args.llm_binding == "azure_openai"
            else openai_alike_model_complete,
            embedding_func=embedding_func,
        )

How would the information be transfered to openai_alike_model_complete ?

[danielaskdd]

I submitted a PR to fix this issue. The OpenAI client only requires specific parameters - passing additional kwargs may cause errors.

#628

[ParisNeo]

Hi I took a look at the pull request but I think this still won't fix it as you don't send the host address nor api key.
I would add:

llm_model_kwargs={
                "host": args.llm_binding_host,
                "api_key": args.llm_binding_api_key,
            },

Although I don't really understand why you did separate the two. kwargs can contains arguments that are not supported by the function without causing any problem. I think this was why those parameters were passed with kwargs and not directly. THis gives the advantage that you can pass parameters even if the current llm don't support them.

[danielaskdd]

I just test the api server with Deepseek and OpenAI LLM, it works just fine for me.

For openai the llm_model_func pass to rag is redefine in lightrag_server.py as:

    async def openai_alike_model_complete(
        prompt,
        system_prompt=None,
        history_messages=[],
        keyword_extraction=False,
        **kwargs,
    ) -> str:
        return await openai_complete_if_cache(
            args.llm_model,
            prompt,
            system_prompt=system_prompt,
            history_messages=history_messages,
            base_url=args.llm_binding_host,
            api_key=args.llm_binding_api_key,
            **kwargs,
        )

The base_url and api_key has already setup there, no need to put it in kwargs. What's more, openai use base_url instead of host for endpoint url. So if you put host in kwargs, openai will complain it, cause llm.py pass kwargs to openai:

       response = await openai_async_client.chat.completions.create(
            model=model, messages=messages, **kwargs
        )

I guess the reason the program not works in your environment may be the .env file or command line params is not up to-date. Here is a working .env file:

# RAG Configuration
MAX_ASYNC=4
MAX_TOKENS=32768
EMBEDDING_DIM=1024
MAX_EMBED_TOKENS=8192

# LLM Configuration (Use valid host. For local services, you can use host.docker.internal)
LLM_BINDING=openai
LLM_MODEL=deepseek-chat
LLM_BINDING_HOST=https://api.deepseek.com
LLM_BINDING_API_KEY=sk-my-api-key-for-deepseek

# if LLM_BINDING_API_KEY missing, OPENAI_API_KEY will work as well
OPENAI_API_KEY=sk-my-api-key-for-openai

# Embedding Configuration (Use valid host. For local services, you can use host.docker.internal)
# Ollama example
EMBEDDING_BINDING=ollama
EMBEDDING_BINDING_HOST=http://localhost:11434
EMBEDDING_MODEL=bge-m3:latest

[ParisNeo]

Thanks alot. I'll take a look.

Nice collaborating with you.

By the way. Are you interested in adding an OpenAI compatible lightrag server? As of today, there is an ollama version, but we can make an option to select ollama or open ai style.

[danielaskdd]

Providing an OpenAI-compatible API for LightRAG is a good idea. It's recommended to use different endpoints to distinguish between OpenAI and ollama APIs, which would allow users to access both interfaces. However, I'm not yet familiar with OpenAI's API specifications, so implementing this may take some time.

[YanSte]

Is this solve ?

[danielaskdd]

Already solved, close it pls.