From 4cc120c7443cf9dab898736f3c3b45dc8f14672b Mon Sep 17 00:00:00 2001 From: Daniel Bevenius Date: Fri, 12 Apr 2024 14:11:46 +0200 Subject: [PATCH] infill : add download instructions for model (#6626) * infill : add download instructions for model This commit adds instructions on how to download a CodeLlama model using the `hf.sh` script. This will download the model and place it in the `models` directory which is the same model use later by the infill example. Signed-off-by: Daniel Bevenius * squash! infill : add download instructions for model Clarify the reason for using CodeLlama. Signed-off-by: Daniel Bevenius --------- Signed-off-by: Daniel Bevenius --- examples/infill/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/examples/infill/README.md b/examples/infill/README.md index 8c97f719b00a0..6b076c8390abf 100644 --- a/examples/infill/README.md +++ b/examples/infill/README.md @@ -36,6 +36,11 @@ The `infill` program offers a seamless way to interact with LLaMA models, allowi ### Example +Download a model that supports infill, for example CodeLlama: +```console +scripts/hf.sh --repo TheBloke/CodeLlama-13B-GGUF --file codellama-13b.Q5_K_S.gguf --outdir models +``` + ```bash ./infill -t 10 -ngl 0 -m models/codellama-13b.Q5_K_S.gguf -c 4096 --temp 0.7 --repeat_penalty 1.1 -n 20 --in-prefix "def helloworld():\n print(\"hell" --in-suffix "\n print(\"goodbye world\")\n " ```