Chapter-10/test-of-llama-2-quantized-with-llama-cpp-on-cpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7ac1208",
   "metadata": {
    "papermill": {
     "duration": 0.003659,
     "end_time": "2023-09-29T12:25:24.327282",
     "exception": false,
     "start_time": "2023-09-29T12:25:24.323623",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction\n",
    "\n",
    "\n",
    "\n",
    "We tested previously Llama v2 model, quantized using llama.cpp, in a separate notebook: https://www.kaggle.com/code/gpreda/fast-test-of-llama-v2-pre-quantized-with-llama-cpp. In that notebook we also describe the process\n",
    "\n",
    "Here we will use CPU to run the quantized model. In a similar "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a2e1a48",
   "metadata": {
    "papermill": {
     "duration": 0.002727,
     "end_time": "2023-09-29T12:25:24.335307",
     "exception": false,
     "start_time": "2023-09-29T12:25:24.332580",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Let's run it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e1bc040",
   "metadata": {
    "papermill": {
     "duration": 0.002454,
     "end_time": "2023-09-29T12:25:24.340506",
     "exception": false,
     "start_time": "2023-09-29T12:25:24.338052",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's first pip install llama.cpp.python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "afc0e86e",
   "metadata": {
    "_kg_hide-input": true,
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:25:24.347818Z",
     "iopub.status.busy": "2023-09-29T12:25:24.347317Z",
     "iopub.status.idle": "2023-09-29T12:26:23.872547Z",
     "shell.execute_reply": "2023-09-29T12:26:23.871102Z"
    },
    "papermill": {
     "duration": 59.532092,
     "end_time": "2023-09-29T12:26:23.875323",
     "exception": false,
     "start_time": "2023-09-29T12:25:24.343231",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting llama-cpp-python\r\n",
      "  Downloading llama_cpp_python-0.2.7.tar.gz (1.6 MB)\r\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m16.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25h  Installing build dependencies ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \bdone\r\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\r\n",
      "\u001b[?25h  Installing backend dependencies ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \bdone\r\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25l-\b \bdone\r\n",
      "\u001b[?25hRequirement already satisfied: typing-extensions>=4.5.0 in /opt/conda/lib/python3.10/site-packages (from llama-cpp-python) (4.6.3)\r\n",
      "Requirement already satisfied: numpy>=1.20.0 in /opt/conda/lib/python3.10/site-packages (from llama-cpp-python) (1.23.5)\r\n",
      "Collecting diskcache>=5.6.1 (from llama-cpp-python)\r\n",
      "  Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)\r\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hBuilding wheels for collected packages: llama-cpp-python\r\n",
      "  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \bdone\r\n",
      "\u001b[?25h  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.2.7-cp310-cp310-manylinux_2_31_x86_64.whl size=982567 sha256=a806ae90bc6d31602b4adbcf3b25e7b11b23952d5868be5942a25e24edd89662\r\n",
      "  Stored in directory: /root/.cache/pip/wheels/76/74/f5/9f64ca4c6ff4c437f5566f52cfbc233660156f74edc3f1e5ec\r\n",
      "Successfully built llama-cpp-python\r\n",
      "Installing collected packages: diskcache, llama-cpp-python\r\n",
      "Successfully installed diskcache-5.6.3 llama-cpp-python-0.2.7\r\n",
      "Cloning into 'llama.cpp'...\r\n",
      "remote: Enumerating objects: 9400, done.\u001b[K\r\n",
      "remote: Counting objects: 100% (9398/9398), done.\u001b[K\r\n",
      "remote: Compressing objects: 100% (2922/2922), done.\u001b[K\r\n",
      "remote: Total 9400 (delta 6585), reused 9106 (delta 6416), pack-reused 2\u001b[K\r\n",
      "Receiving objects: 100% (9400/9400), 8.57 MiB | 19.90 MiB/s, done.\r\n",
      "Resolving deltas: 100% (6585/6585), done.\r\n"
     ]
    }
   ],
   "source": [
    "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" pip install llama-cpp-python\n",
    "!git clone https://github.com/ggerganov/llama.cpp.git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88ac17f",
   "metadata": {
    "papermill": {
     "duration": 0.006538,
     "end_time": "2023-09-29T12:26:23.889442",
     "exception": false,
     "start_time": "2023-09-29T12:26:23.882904",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Then, we import from the llama_cpp package the Llama module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bd1be71e",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:26:23.905284Z",
     "iopub.status.busy": "2023-09-29T12:26:23.904885Z",
     "iopub.status.idle": "2023-09-29T12:26:23.961544Z",
     "shell.execute_reply": "2023-09-29T12:26:23.960359Z"
    },
    "papermill": {
     "duration": 0.067761,
     "end_time": "2023-09-29T12:26:23.964270",
     "exception": false,
     "start_time": "2023-09-29T12:26:23.896509",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_cpp import Llama\n",
    "from time import time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645fe601",
   "metadata": {
    "papermill": {
     "duration": 0.006715,
     "end_time": "2023-09-29T12:26:23.978196",
     "exception": false,
     "start_time": "2023-09-29T12:26:23.971481",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Load the quantized model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b3230a7e",
   "metadata": {
    "_kg_hide-input": true,
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:26:23.993748Z",
     "iopub.status.busy": "2023-09-29T12:26:23.993144Z",
     "iopub.status.idle": "2023-09-29T12:27:41.027034Z",
     "shell.execute_reply": "2023-09-29T12:27:41.025843Z"
    },
    "papermill": {
     "duration": 77.054002,
     "end_time": "2023-09-29T12:27:41.039084",
     "exception": false,
     "start_time": "2023-09-29T12:26:23.985082",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "llama_model_loader: loaded meta data with 18 key-value pairs and 291 tensors from /kaggle/input/test-llama-2-quantized-with-llama-cpp/llama-7b.gguf (version GGUF V2 (latest))\n",
      "llama_model_loader: - tensor    0:                token_embd.weight q8_0     [  4096, 32000,     1,     1 ]\n",
      "llama_model_loader: - tensor    1:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor    2:            blk.0.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor    3:            blk.0.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor    4:              blk.0.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor    5:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor    6:              blk.0.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor    7:         blk.0.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor    8:              blk.0.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor    9:              blk.0.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   10:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   11:            blk.1.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   12:            blk.1.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   13:              blk.1.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   14:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   15:              blk.1.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   16:         blk.1.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   17:              blk.1.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   18:              blk.1.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   19:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   20:           blk.10.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   21:           blk.10.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   22:             blk.10.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   23:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   24:             blk.10.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   25:        blk.10.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   26:             blk.10.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   27:             blk.10.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   28:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   29:           blk.11.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   30:           blk.11.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   31:             blk.11.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   32:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   33:             blk.11.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   34:        blk.11.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   35:             blk.11.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   36:             blk.11.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   37:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   38:           blk.12.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   39:           blk.12.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   40:             blk.12.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   41:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   42:             blk.12.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   43:        blk.12.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   44:             blk.12.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   45:             blk.12.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   46:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   47:           blk.13.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   48:           blk.13.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   49:             blk.13.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   50:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   51:             blk.13.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   52:        blk.13.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   53:             blk.13.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   54:             blk.13.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   55:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   56:           blk.14.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   57:           blk.14.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   58:             blk.14.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   59:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   60:             blk.14.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   61:        blk.14.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   62:             blk.14.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   63:             blk.14.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   64:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   65:           blk.15.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   66:           blk.15.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   67:             blk.15.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   68:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   69:             blk.15.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   70:        blk.15.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   71:             blk.15.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   72:             blk.15.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   73:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   74:           blk.16.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   75:           blk.16.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   76:             blk.16.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   77:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   78:             blk.16.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   79:        blk.16.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   80:             blk.16.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   81:             blk.16.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   82:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   83:           blk.17.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   84:           blk.17.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   85:             blk.17.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   86:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   87:             blk.17.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   88:        blk.17.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   89:             blk.17.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   90:             blk.17.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   91:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   92:           blk.18.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   93:           blk.18.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   94:             blk.18.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor   95:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor   96:             blk.18.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   97:        blk.18.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   98:             blk.18.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor   99:             blk.18.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  100:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  101:           blk.19.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  102:           blk.19.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  103:             blk.19.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  104:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  105:             blk.19.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  106:        blk.19.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  107:             blk.19.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  108:             blk.19.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  109:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  110:            blk.2.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  111:            blk.2.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  112:              blk.2.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  113:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  114:              blk.2.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  115:         blk.2.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  116:              blk.2.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  117:              blk.2.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  118:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  119:           blk.20.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  120:           blk.20.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  121:             blk.20.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  122:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  123:             blk.20.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  124:        blk.20.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  125:             blk.20.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  126:             blk.20.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  127:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  128:           blk.21.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  129:           blk.21.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  130:             blk.21.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  131:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  132:             blk.21.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  133:        blk.21.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  134:             blk.21.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  135:             blk.21.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  136:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  137:           blk.22.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  138:           blk.22.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  139:             blk.22.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  140:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  141:             blk.22.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  142:        blk.22.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  143:             blk.22.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  144:             blk.22.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  145:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  146:           blk.23.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  147:           blk.23.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  148:             blk.23.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  149:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  150:             blk.23.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  151:        blk.23.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  152:             blk.23.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  153:             blk.23.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  154:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  155:            blk.3.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  156:            blk.3.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  157:              blk.3.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  158:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  159:              blk.3.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  160:         blk.3.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  161:              blk.3.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  162:              blk.3.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  163:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  164:            blk.4.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  165:            blk.4.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  166:              blk.4.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  167:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  168:              blk.4.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  169:         blk.4.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  170:              blk.4.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  171:              blk.4.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  172:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  173:            blk.5.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  174:            blk.5.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  175:              blk.5.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  176:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  177:              blk.5.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  178:         blk.5.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  179:              blk.5.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  180:              blk.5.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  181:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  182:            blk.6.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  183:            blk.6.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  184:              blk.6.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  185:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  186:              blk.6.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  187:         blk.6.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  188:              blk.6.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  189:              blk.6.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  190:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  191:            blk.7.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  192:            blk.7.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  193:              blk.7.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  194:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  195:              blk.7.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  196:         blk.7.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  197:              blk.7.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  198:              blk.7.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  199:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  200:            blk.8.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  201:            blk.8.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  202:              blk.8.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  203:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  204:              blk.8.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  205:         blk.8.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  206:              blk.8.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  207:              blk.8.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  208:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  209:            blk.9.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  210:            blk.9.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  211:              blk.9.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  212:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  213:              blk.9.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  214:         blk.9.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  215:              blk.9.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  216:              blk.9.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  217:                    output.weight q8_0     [  4096, 32000,     1,     1 ]\n",
      "llama_model_loader: - tensor  218:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  219:           blk.24.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  220:           blk.24.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  221:             blk.24.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  222:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  223:             blk.24.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  224:        blk.24.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  225:             blk.24.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  226:             blk.24.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  227:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  228:           blk.25.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  229:           blk.25.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  230:             blk.25.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  231:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  232:             blk.25.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  233:        blk.25.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  234:             blk.25.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  235:             blk.25.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  236:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  237:           blk.26.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  238:           blk.26.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  239:             blk.26.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  240:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  241:             blk.26.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  242:        blk.26.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  243:             blk.26.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  244:             blk.26.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  245:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  246:           blk.27.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  247:           blk.27.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  248:             blk.27.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  249:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  250:             blk.27.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  251:        blk.27.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  252:             blk.27.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  253:             blk.27.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  254:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  255:           blk.28.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  256:           blk.28.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  257:             blk.28.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  258:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  259:             blk.28.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  260:        blk.28.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  261:             blk.28.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  262:             blk.28.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  263:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  264:           blk.29.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  265:           blk.29.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  266:             blk.29.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  267:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  268:             blk.29.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  269:        blk.29.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  270:             blk.29.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  271:             blk.29.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  272:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  273:           blk.30.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  274:           blk.30.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  275:             blk.30.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  276:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  277:             blk.30.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  278:        blk.30.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  279:             blk.30.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  280:             blk.30.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  281:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  282:           blk.31.ffn_down.weight q8_0     [ 11008,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  283:           blk.31.ffn_gate.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  284:             blk.31.ffn_up.weight q8_0     [  4096, 11008,     1,     1 ]\n",
      "llama_model_loader: - tensor  285:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - tensor  286:             blk.31.attn_k.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  287:        blk.31.attn_output.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  288:             blk.31.attn_q.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  289:             blk.31.attn_v.weight q8_0     [  4096,  4096,     1,     1 ]\n",
      "llama_model_loader: - tensor  290:               output_norm.weight f32      [  4096,     1,     1,     1 ]\n",
      "llama_model_loader: - kv   0:                       general.architecture str     \n",
      "llama_model_loader: - kv   1:                               general.name str     \n",
      "llama_model_loader: - kv   2:                       llama.context_length u32     \n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32     \n",
      "llama_model_loader: - kv   4:                          llama.block_count u32     \n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32     \n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32     \n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32     \n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32     \n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32     \n",
      "llama_model_loader: - kv  10:                          general.file_type u32     \n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str     \n",
      "llama_model_loader: - kv  12:                      tokenizer.ggml.tokens arr     \n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.scores arr     \n",
      "llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr     \n",
      "llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32     \n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32     \n",
      "llama_model_loader: - kv  17:            tokenizer.ggml.unknown_token_id u32     \n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q8_0:  226 tensors\n",
      "llm_load_print_meta: format         = GGUF V2 (latest)\n",
      "llm_load_print_meta: arch           = llama\n",
      "llm_load_print_meta: vocab type     = SPM\n",
      "llm_load_print_meta: n_vocab        = 32000\n",
      "llm_load_print_meta: n_merges       = 0\n",
      "llm_load_print_meta: n_ctx_train    = 2048\n",
      "llm_load_print_meta: n_ctx          = 512\n",
      "llm_load_print_meta: n_embd         = 4096\n",
      "llm_load_print_meta: n_head         = 32\n",
      "llm_load_print_meta: n_head_kv      = 32\n",
      "llm_load_print_meta: n_layer        = 32\n",
      "llm_load_print_meta: n_rot          = 128\n",
      "llm_load_print_meta: n_gqa          = 1\n",
      "llm_load_print_meta: f_norm_eps     = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps = 1.0e-05\n",
      "llm_load_print_meta: n_ff           = 11008\n",
      "llm_load_print_meta: freq_base      = 10000.0\n",
      "llm_load_print_meta: freq_scale     = 1\n",
      "llm_load_print_meta: model type     = 7B\n",
      "llm_load_print_meta: model ftype    = mostly Q8_0\n",
      "llm_load_print_meta: model params   = 6.74 B\n",
      "llm_load_print_meta: model size     = 6.67 GiB (8.50 BPW) \n",
      "llm_load_print_meta: general.name   = 7b-chat-hf\n",
      "llm_load_print_meta: BOS token = 1 '<s>'\n",
      "llm_load_print_meta: EOS token = 2 '</s>'\n",
      "llm_load_print_meta: UNK token = 0 '<unk>'\n",
      "llm_load_print_meta: LF token  = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size =    0.09 MB\n",
      "llm_load_tensors: mem required  = 6828.73 MB (+  256.00 MB per state)\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: kv self size  =  256.00 MB\n",
      "llama_new_context_with_model: compute buffer total size =   71.97 MB\n",
      "AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | \n"
     ]
    }
   ],
   "source": [
    "t1 = time()\n",
    "llm = Llama(model_path=\"/kaggle/input/test-llama-2-quantized-with-llama-cpp/llama-7b.gguf\")\n",
    "t2 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "86d15c60",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:27:41.061022Z",
     "iopub.status.busy": "2023-09-29T12:27:41.060199Z",
     "iopub.status.idle": "2023-09-29T12:27:41.066647Z",
     "shell.execute_reply": "2023-09-29T12:27:41.065305Z"
    },
    "papermill": {
     "duration": 0.021858,
     "end_time": "2023-09-29T12:27:41.068945",
     "exception": false,
     "start_time": "2023-09-29T12:27:41.047087",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load the model: 77.028 sec.\n"
     ]
    }
   ],
   "source": [
    "print(f\"Time to load the model: {round(t2 - t1, 3)} sec.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ebf0546",
   "metadata": {
    "papermill": {
     "duration": 0.007704,
     "end_time": "2023-09-29T12:27:41.085283",
     "exception": false,
     "start_time": "2023-09-29T12:27:41.077579",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And now, let's just try a prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "61c9394f",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:27:41.103324Z",
     "iopub.status.busy": "2023-09-29T12:27:41.102237Z",
     "iopub.status.idle": "2023-09-29T12:27:54.016429Z",
     "shell.execute_reply": "2023-09-29T12:27:54.015566Z"
    },
    "papermill": {
     "duration": 12.925271,
     "end_time": "2023-09-29T12:27:54.018385",
     "exception": false,
     "start_time": "2023-09-29T12:27:41.093114",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =  4145.28 ms\n",
      "llama_print_timings:      sample time =    12.97 ms /    18 runs   (    0.72 ms per token,  1387.82 tokens per second)\n",
      "llama_print_timings: prompt eval time =  4145.19 ms /    13 tokens (  318.86 ms per token,     3.14 tokens per second)\n",
      "llama_print_timings:        eval time =  8688.99 ms /    17 runs   (  511.12 ms per token,     1.96 tokens per second)\n",
      "llama_print_timings:       total time = 12906.40 ms\n"
     ]
    }
   ],
   "source": [
    "t1 = time()\n",
    "output = llm(\"Q: Name three capital cities in Europe? A: \", \n",
    "             max_tokens=38, \n",
    "             stop=[\"Q:\", \"\\n\"], \n",
    "             echo=True)\n",
    "t2 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f0e2e7db",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:27:54.036628Z",
     "iopub.status.busy": "2023-09-29T12:27:54.035628Z",
     "iopub.status.idle": "2023-09-29T12:27:54.040930Z",
     "shell.execute_reply": "2023-09-29T12:27:54.039929Z"
    },
    "papermill": {
     "duration": 0.016668,
     "end_time": "2023-09-29T12:27:54.043153",
     "exception": false,
     "start_time": "2023-09-29T12:27:54.026485",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to run the query: 12.909 sec.\n"
     ]
    }
   ],
   "source": [
    "print(f\"Time to run the query: {round(t2 - t1, 3)} sec.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0782b79",
   "metadata": {
    "papermill": {
     "duration": 0.007654,
     "end_time": "2023-09-29T12:27:54.058877",
     "exception": false,
     "start_time": "2023-09-29T12:27:54.051223",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "And now let's inspect the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "673ae058",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2023-09-29T12:27:54.077560Z",
     "iopub.status.busy": "2023-09-29T12:27:54.076942Z",
     "iopub.status.idle": "2023-09-29T12:27:54.084814Z",
     "shell.execute_reply": "2023-09-29T12:27:54.083702Z"
    },
    "papermill": {
     "duration": 0.020638,
     "end_time": "2023-09-29T12:27:54.087473",
     "exception": false,
     "start_time": "2023-09-29T12:27:54.066835",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'cmpl-8d695f81-5d49-4c5b-a727-ebd72d2f9375',\n",
       " 'object': 'text_completion',\n",
       " 'created': 1695990461,\n",
       " 'model': '/kaggle/input/test-llama-2-quantized-with-llama-cpp/llama-7b.gguf',\n",
       " 'choices': [{'text': 'Q: Name three capital cities in Europe? A: 1. Berlin, Germany 2. Paris, France 3. London, England',\n",
       "   'index': 0,\n",
       "   'logprobs': None,\n",
       "   'finish_reason': 'stop'}],\n",
       " 'usage': {'prompt_tokens': 13, 'completion_tokens': 17, 'total_tokens': 30}}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e937c930",
   "metadata": {
    "papermill": {
     "duration": 0.007804,
     "end_time": "2023-09-29T12:27:54.103604",
     "exception": false,
     "start_time": "2023-09-29T12:27:54.095800",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "That's all. Try it out yourself."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 154.552516,
   "end_time": "2023-09-29T12:27:55.536944",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-09-29T12:25:20.984428",
   "version": "2.4.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}