Skip to content

Commit

Permalink
cleaned up how to guide
Browse files Browse the repository at this point in the history
  • Loading branch information
musoles committed Sep 24, 2024
1 parent f370918 commit e99a046
Showing 1 changed file with 1 addition and 17 deletions.
18 changes: 1 addition & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ $ kalavai job list
└───────────────────┴───────────────────────────────────┴────────────────────────┘
```

Kalavai creates an endpoint for each deployed job, which is displayed above. In the case of vLLM jobs, this is a model endpoint that can be interacted as you would any [LLM server](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#using-openai-completions-api-with-vllm). For example:
Kalavai creates an endpoint for each deployed job, which is displayed above. In the case of vLLM jobs, this is a model endpoint that can be interacted as you would any [LLM server](https://docs.vllm.ai/en/latest/getting_started/quickstart.html#using-openai-completions-api-with-vllm). See the [vLLM template documentation](templates/vllm/README.md) for info on how to interact with the model, but as a quick go:
```bash
curl http://100.8.0.2:31992/v1/completions \
-H "Content-Type: application/json" \
Expand All @@ -172,22 +172,6 @@ curl http://100.8.0.2:31992/v1/completions \
}'
```

Also from python:
```python
from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://100.8.0.2:31992/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
completion = client.completions.create(model="facebook/opt-350m",
prompt="San Francisco is a")
print("Completion result:", completion)
```

For more information on what a template can do:
```bash
kalavai job describe vllm
Expand Down

0 comments on commit e99a046

Please sign in to comment.