Running great on AMD RX6800 XT

[vampyre2000]

Just a big thankyou for adding support for AMD cards.
The performance of this software is wonderful and has made running local LLMs practical on AMD hardware.
Should of mentioned system specs.
Windows 11
AMD RX6800 XT
AMD Adrenaline edition: 24.1.1
AMD Ryzen 5 5600
32 GB RAM

[eldr4]

I'd also like to add my appreciation to this. Also working great on my 6700 XT.

[Vamsi963]

working fine in 6700xt on windows 11. Is there a way i can use it in python

[vampyre2000]

You run the program as normal and it will start the server.
KoboldCPP dafaults to http://localhost:5001/
Then you just need to connect to the server API via python code.
See example code, below, there are other ways to do this too, but this is one easy way.
'''
from openai import OpenAI
client = OpenAI(base_url="http://localhost:5001/v1", api_key="not-needed")
completion = client.chat.completions.create(
model="local-model", # this field is currently unused
messages=[
{"role": "system", "content": "Always answer in rhymes."},
{"role": "user", "content": "Introduce yourself."}
],
temperature=0.8,
)
print(completion.choices[0].message)
'''

[ramzeez88]

Just a big thankyou for adding support for AMD cards. The performance of this software is wonderful and has made running local LLMs practical on AMD hardware. Should of mentioned system specs. Windows 11 AMD RX6800 XT AMD Adrenaline edition: 24.1.1 AMD Ryzen 5 5600 32 GB RAM

Hi, what sort of speeds do you get ?