server : (webui) revamp Settings dialog, add Pyodide interpreter

[ngxson]

In this PR:

• revamp Settings dialog, make it 2 columns
• add the "Experimentals" section, currently having "Python interpreter"
• add API for side panel, aka "Canvas" that can be extended to support other type of canvas in the future, the UX should be like Claude's canvas or ChatGPT's artifacts

The python interpreter uses pyodide under the hood, which is CPython compiled as webassembly. Due to big bundle size, this feature requires internet connection to download the pyodide and wasm file from CDN.

Test with a real world problem

[ggerganov]

Very cool stuff.

It seems to not handle errors atm:

[ngxson]

Ok thanks for testing, the exception should be handled now!

[ggerganov]

How did you get it to generate the "Mortgage Calculator" title? Qwen Coder does not seem to do it.

[ngxson]

How did you get it to generate the "Mortgage Calculator" title? Qwen Coder does not seem to do it.

I'm using llama 3.1 8B, but it's quite random, it never does that again when I regenerate the message

[ggerganov]

Maybe there should be a way to stop execution of long programs:

[ngxson]

Maybe there should be a way to stop execution of long programs:

Yeah right, it's also blocking main thread so the UX is not very good. But unfortunately it's quite complicated to fix, because this requires implementing a web worker: https://pyodide.org/en/stable/usage/webworker.html

I'll see if I can fix in this PR, but otherwise I think we can still release the first version without it, to be added later on.

[woof-dog]

Is there a way we can have llama-server serve these dependencies so the internet connection is not required? Like, some folder we can dump them into.

Maybe a future issue but would prefer to serve these directly

[ngxson]

Is there a way we can have llama-server serve these dependencies so the internet connection is not required

We can and it's trivial to do. But I'm not comfortable of shipping a binary containing 8MB of llama.cpp, 1MB of webui and 11MB of pyodide, it's literally a bloatware to many users who only uses llama-server via API.

[MoonRide303]

Is there a way we can have llama-server serve these dependencies so the internet connection is not required

We can and it's trivial to do. But I'm not comfortable of shipping a binary containing 8MB of llama.cpp, 1MB of webui and 11MB of pyodide, it's literally a bloatware to many users who only uses llama-server via API.

Both points sound valid - local servers should be fully usable without connecting to the Internet, but keeping the core as small as possible is important, too. If it's a simple thing, then maybe you could just add some brief instructions how to run it this way?

[ngxson]

@MoonRide303 it's trivial to implement doen't mean end-user can easily do that. What I mean by trivial is that we as developer can download the vendor library to /public directory and compile llama-server with it.

A better way would be to do this completely on the web level and not at cpp level.

We can download the vendor library (in this case, pyodide wasm file) into browser's cache and store it there. User only need to download once, then next time we load from browser's cache.

Would appreciate if someone can play with this.

[woof-dog]

I agree with @MoonRide303 and that was my original intention. For the average user they can download it from the pyodide website, store it in their browser's cache, whatever. For power users that want 100% control over the longevity/accessibility/privacy of their service and want to self host their llama.cpp with no outside dependencies (after setup) that can run forever, we should have some "somewhat easy" way for them to do that without having to modify code themselves.

For example, maybe we can hint which URL to load pyodide from in some server endpoint, where an absent value means just use the default, but if the user wants to locally serve pyodide they can change it.