examples/01.Getting_Started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/merantix-momentum/squirrel-datasets-core/blob/main/examples/01.Getting_Started.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eN4eI8-L64v_",
    "tags": [
     "skip-execution"
    ]
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import matplotlib\n",
    "    import squirrel\n",
    "    import squirrel_datasets_core\n",
    "\n",
    "    restart_colab = False\n",
    "except:\n",
    "    !pip install -q --ignore-requires-python --upgrade \"squirrel-datasets-core[torchvision]\" matplotlib # noqa\n",
    "    import matplotlib\n",
    "    import squirrel\n",
    "    import squirrel_datasets_core\n",
    "\n",
    "    restart_colab = True\n",
    "\n",
    "print(squirrel.__version__)\n",
    "print(squirrel_datasets_core.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run this tutorial in google colab, there unfortunately are some tweaks we need to apply to make it work - we will skip this step automatically if we cannot detect colab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "\n",
    "    if restart_colab:\n",
    "        !pip install -q --upgrade --force pyyaml==5.4.1\n",
    "\n",
    "        # need to restart kernel\n",
    "        import os\n",
    "\n",
    "        os.kill(os.getpid(), 9)\n",
    "except:\n",
    "    # not in colab\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Squirrel enables you to efficiently load existing datasets and also share datasets using `Catalog` and `Driver`,\n",
    "apply transformations in a performant and scalable way using `Iterstream`,\n",
    "and store datasets in a way that is most convenient for deep learning applications with `SquirrelStore`.\n",
    "\n",
    "Let's see an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y4IIgrUe7CgJ"
   },
   "outputs": [],
   "source": [
    "from squirrel.catalog import Catalog\n",
    "\n",
    "# init catalog with in-built datasets\n",
    "cat = Catalog.from_plugins()\n",
    "\n",
    "# access training images from CIFAR-10 dataset\n",
    "driver = cat[\"cifar10\"].get_driver()\n",
    "it = driver.get_iter()\n",
    "\n",
    "# retrieve single sample from CIFAR-10\n",
    "sample = it.take(1).collect()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YqceUBWF7gBC"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# plot sample and label\n",
    "plt.title(f\"Class: {sample[1]}\")\n",
    "plt.imshow(sample[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we use `Catalog` to load `CIFAR-10` dataset.\n",
    "`driver.get_iter()` gives us a `Composable` object from the package `Iterstream`, which enables us to apply transformations using convenient methods such as `map` and `filter`.\n",
    "\n",
    "Let's see an example of these transformations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from squirrel.iterstream import IterableSource\n",
    "\n",
    "it = IterableSource([1, 2, 3]).map(lambda x: x + 1).async_map(lambda x: x**2).filter(lambda x: x % 2 == 0)\n",
    "for i in it:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Iterstream` can scale up or out your data loading using asynchronous execution using a local executor or a dask cluster, and just-in-time compilation using numba.\n",
    "Be sure to checkout the documentation."
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 4
}