Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flake app to run openai proxy #4612

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add flake app to run openai proxy #4612

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions flake.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,15 @@
type = "app";
program = "${self.packages.${system}.default}/bin/llama-server";
};
apps.llama-server-openai-proxy = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to mention this in #4605, but we wouldn't have had to manually wrap these in apps if we instead exposed them (in the overlay and) in packages/legacyPackages, and if we ensured that meta.mainPrograms are set correctly (with writeShellScriptBin they are)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could now add this to scope.nix:

llama.cpp/.devops/nix/scope.nix

Lines 7 to 12 in 4e1d251

lib.makeScope newScope (
self: {
inherit llamaVersion;
llama-cpp = self.callPackage ./package.nix { };
}
)

This way the attribute would show up as llamaPackages.llama-server-openai-proxy in nixpkgs with the flake's overlays.default applied, as well as in the flake's legacyPackages.${system}.llamaPackages.

type = "app";
program = "${
(let pythonWithPkgs = pkgs.python3.withPackages (ps: with ps; [ flask requests ]);
in pkgs.writeShellScriptBin "run-openai-proxy" ''
${pythonWithPkgs}/bin/python3 ${self}/examples/server/api_like_OAI.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could've been a makeWrapper. I don't know if api_like_OAI.py takes any command line arguments, but makeWrapper would have passed those on

'')
}/bin/run-openai-proxy";
Comment on lines +116 to +120
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't suggest anything concrete right now, but it might be a good thing to do some form of buildPythonApplication because its probably less affected by PYTHONPATH contamination.

};
apps.llama-embedding = {
type = "app";
program = "${self.packages.${system}.default}/bin/embedding";
Expand Down