[Bug]: Error after deleting fallback using UI

[vargacypher]

What happened?

We're testing litllm proxy server to helped us with fallbacks, and we're facing a problem when trying to delete a fallback rule and recreating it.

• Docker deployment using litellm:main-latest
• Postgres DB

config.yaml

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY     
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY   
  - model_name: gemini-pro
    litellm_params:
      model: vertex_ai/gemini-2.0-flash-001 
      vertex_project: "project"
      vertex_location: "us-central1"
      vertex_credentials: "/app/user.json" 

How to replicate:

In UI panel >> router settings >> add falbacks >> test

Success

In UI panel >> router settings >> delete falbacks >> add (updated) fallback >> test

Error

Relevant log output

Error occurred while generating model response. Please try again. Error: Error: 500 litellm.InternalServerError: This is a mock exception for model=gemini-pro, to trigger a fallback. Fallbacks=[{}, {'gemini-pro': ['gpt-4o-mini']}] Received Model Group=gemini-pro Available Model Group Fallbacks=None Error doing the fallback: list index out of range

Are you a ML Ops Team?

Yes

What LiteLLM version are you on ?

latest

Twitter / LinkedIn details

No response

[krrishdholakia]

Thank you for this issue @vargacypher i'll pick this up tomorrow.

[rzqtet]

For me to fix it I needed to open LiteLLM_Config table in the database and add it manually

[krrishdholakia]

cc: @ishaan-jaff for qa

[krrishdholakia]

Picking this up today

[dkindlund]

Yeah, deleting and re-adding fallback model groups via the Admin UI doesn't work either -- you'll just end up creating more empty placeholders in the array stored in the DB:

prod-litellm-conf=> select param_value from "LiteLLM_Config" where param_name = 'router_settings';
                                                                                                                                param_value
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 {"timeout": 6000.0, "fallbacks": [{}, {}], "num_retries": 3, "retry_after": 5.0, "allowed_fails": 1, "cooldown_time": 30.0, "routing_strategy": "latency-based-routing", "routing_strategy_args": {"ttl": 3600, "lowest_latency_buffer": 0, "max_latency_list_size": 10}}
(1 row)

This part is clearly the bug:

"fallbacks": [{}, {}]