06-linq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LINQ\n",
    "\n",
    "LINQ stands for Language Integrated Queries. LINQ is a set extensions methods and some special C# syntax that simplifies the work with collections.\n",
    "\n",
    "LINQ allows to do things like filtering items in collection, projecting them to new forms or aggregating items in a concise syntax. All of these things can also be accomplished using simple loops and temporary variables, but the need to do that in so frequent, that it warrants the need of a standard library for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Sample data set to be used throughout the notebook\n",
    "\n",
    "class Person\n",
    "{\n",
    "    public string FirstName { get; set; }\n",
    "    public string LastName { get; set; }\n",
    "    public int Age { get; set; }\n",
    "}\n",
    "\n",
    "var data = new List<Person>\n",
    "{\n",
    "    new Person { FirstName = \"John\", LastName = \"Doe\", Age = 25 },\n",
    "    new Person { FirstName = \"Jane\", LastName = \"Doe\", Age = 26 },\n",
    "    new Person { FirstName = \"John\", LastName = \"Smith\", Age = 30 },\n",
    "    new Person { FirstName = \"Jane\", LastName = \"Smith\", Age = 31 },\n",
    "    new Person { FirstName = \"John\", LastName = \"Johnson\", Age = 35 },\n",
    "    new Person { FirstName = \"Jane\", LastName = \"Johnson\", Age = 36 },\n",
    "    new Person { FirstName = \"John\", LastName = \"Williams\", Age = 40 },\n",
    "    new Person { FirstName = \"Jane\", LastName = \"Williams\", Age = 41 },\n",
    "    new Person { FirstName = \"John\", LastName = \"Brown\", Age = 45 },\n",
    "    new Person { FirstName = \"Jane\", LastName = \"Brown\", Age = 46 },\n",
    "    new Person { FirstName = \"Anthony\", LastName = \"Davis\", Age = 50 },\n",
    "    new Person { FirstName = \"Jessica\", LastName = \"Davis\", Age = 51 },\n",
    "    new Person { FirstName = \"Michael\", LastName = \"Davis\", Age = 55 },\n",
    "    new Person { FirstName = \"Michelle\", LastName = \"Davis\", Age = 56 },\n",
    "    new Person { FirstName = \"David\", LastName = \"Davis\", Age = 60 },\n",
    "    new Person { FirstName = \"Danielle\", LastName = \"Davis\", Age = 61 },\n",
    "    new Person { FirstName = \"Daniel\", LastName = \"Davis\", Age = 65 },\n",
    "    new Person { FirstName = \"Diana\", LastName = \"Davis\", Age = 66 },\n",
    "    new Person { FirstName = \"Dennis\", LastName = \"Davis\", Age = 70 },\n",
    "};"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The problem\n",
    "\n",
    "Let's say we wanted to get all people from dataset, whose first name starts with the letter \"D\". An obvious approach would be to iterate though the collection, run some predicate on the items, and move items that match the predicate to some separate collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var results = new List<Person>();\n",
    "\n",
    "foreach (var person in data)\n",
    "{\n",
    "    if (person.FirstName.StartsWith(\"D\"))\n",
    "    {\n",
    "        results.Add(person);\n",
    "    }\n",
    "}\n",
    "\n",
    "results.DisplayTable();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly and average age post the people in the list can be calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var ageTotal = 0;\n",
    "\n",
    "foreach (var person in data)\n",
    "{\n",
    "    ageTotal += person.Age;\n",
    "}\n",
    "\n",
    "var averageAge = ageTotal / data.Count;\n",
    "\n",
    "averageAge.Display();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or concatenate the names of all the persons in data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "var allNames = String.Empty;\n",
    "\n",
    "foreach (var person in data)\n",
    "{\n",
    "    allNames += $\"{person.FirstName} {person.LastName}, \";\n",
    "}\n",
    "\n",
    "allNames = allNames.TrimEnd(',', ' ');\n",
    "\n",
    "allNames.Display();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The common pattern can be seen:\n",
    "1. Define some variable to accumulate the results.\n",
    "2. Check if the item in the collection matches the predicate.\n",
    "3. Update the result accumulator.\n",
    "\n",
    "Looking at the code, most of it is identical except a few custom bits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the LINQ\n",
    "\n",
    "All of the LINQ methods are generic and will be type safe. This is needed to be able to reliably write predicates or projections.\n",
    "\n",
    "All of the LINQ extension methods reside in `System.Linq` namespace."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.Where()`\n",
    "\n",
    "`.Where()` methods takes in an argument of predicate and returns the list of items in the original collection that matches the given predicate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Same example where we check if the first name starts with \"D\" using LINQ\n",
    "\n",
    "data.Where(p => p.FirstName.StartsWith(\"D\")).DisplayTable();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.Select()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.Select()` methods are used to project data in the collection to different form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Select all names from data collection into concatanated strings\n",
    "\n",
    "data.Select(p => $\"{p.FirstName} {p.LastName}\").DisplayTable();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.Aggregate()`\n",
    "\n",
    "`.Aggregate()` method reduces the number of value in collection into a single one. The resulting value must be of the same type as the original value.\n",
    "\n",
    "This time - `.Aggregate()` takes in a `Func` of 2 arguments as it's argument: 1st argument holds the aggregated value and the 2nd argument holds in current working item from the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Calculate the sum age of all persons in the data collection.\n",
    "// Because the aggregated type must be the same as the source type,\n",
    "// first the values have to be projected to in by Select'ing them.\n",
    "data.Select(x => x.Age).Aggregate((x, y) => x + y).Display();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other aggregates\n",
    "\n",
    "Based on the `.Aggregate()` method, there are various other aggregate-like methods for the most common use cases like sum, average etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Calculate the sum of all ages in the data collection.\n",
    "data.Sum(p => p.Age).Display();\n",
    "\n",
    "// Calculate the average age of all persons in the data collection.\n",
    "data.Average(p => p.Age).Display();\n",
    "\n",
    "// Calculate the maximum age of all persons in the data collection.\n",
    "data.Max(p => p.Age).Display();\n",
    "\n",
    "// Calculate the minimum age of all persons in the data collection.\n",
    "data.Min(p => p.Age).Display();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.Any()` and `.All()`\n",
    "\n",
    "`.Any()` methods returns a boolean that indicates if any of the elements in the collection matches the given predicate. `.All()` methods return boolean that indicates if all the elements in the collection match the given predicate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Check if anyone in the collection is over the age of 90.\n",
    "data.Any(p => p.Age > 90).Display();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Check if everyone in the collection is over the age of 18.\n",
    "data.All(p => p.Age > 18).Display();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Check if everyone in the collection is over the age of 18.\n",
    "data.All(p => p.Age > 18).Display();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Check if everyone in the collection is over the age of 50.\n",
    "data.All(p => p.Age > 50).Display();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `.Group()`\n",
    "\n",
    "`.Group()` allows to group the collection into several subgroups based on the given expression. `.Group`ing the collection will results in a collection of collections, with each of them having an additional `.Key` property that indicates the value by which it was grouped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Group the data by the first name of the person.\n",
    "data.GroupBy(x => x.FirstName).DisplayTable();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Group by, as well as any other LINQ method, can be chained, to get more specific results.\n",
    "data\n",
    "    .GroupBy(x => x.FirstName)\n",
    "    .Select(x => new { FirstName = x.Key, Count = x.Count() })\n",
    "    .OrderByDescending(x => x.Count)\n",
    "    .ThenBy(x => x.FirstName)\n",
    "    .DisplayTable();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting single element out of the collection\n",
    "\n",
    "There are a few options how to select single element out of the collection:\n",
    "- `First()` takes the first element out, throws if there are no elements.\n",
    "- `FirstOrDefault()` takes the first element out or default value for the type of there are no element in the collection.\n",
    "- `Single()` takes the single element out of th collection, throws if the element count in collection is different than 1.\n",
    "- `SingleOrDefault()` takes the single element out of the collection if there is only 1, returns default value for the type if there are none, throws if there are more than 1 element in the collection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// First usage examples\n",
    "Console.WriteLine(data.First());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// First but the collection is empty\n",
    "Console.WriteLine(new List<Person>().First());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// FirstOrDefault on the empty collection\n",
    "Console.WriteLine(new List<Person>().FirstOrDefault());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Single on the empty collection\n",
    "Console.WriteLine(new List<Person>().Single());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// SingleOrDefault on the empty collection\n",
    "Console.WriteLine(new List<Person>().SingleOrDefault());"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// SingleOrDefault on the collection with more than one element\n",
    "Console.WriteLine(data.SingleOrDefault());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `.Take()` and `.Skip()`\n",
    "\n",
    "`.Take()` selects first `n` items from the given collection.\n",
    "\n",
    "Similarly `.Skip()` skips over first `n` element from the collection. it is typically used in conjunction with `.Take()` to implement things like pagination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Skip it and take the first 5 elements from the data collection.\n",
    "\n",
    "var orderedByName = data\n",
    "    .OrderBy(p => p.FirstName)\n",
    "    .ThenBy(p => p.LastName);\n",
    "\n",
    "orderedByName.DisplayTable();\n",
    "\n",
    "var skippedAndTaken = orderedByName\n",
    "    .Skip(2)\n",
    "    .Take(5);\n",
    "\n",
    "skippedAndTaken.DisplayTable();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Query syntax\n",
    "\n",
    "The examples above used the extension method syntax. However C# also has the special query syntax for LINQ usage. It has SQL-like syntax, that is said to be more verbose and self explanatory than extension method syntax. The assumption is that it should be easier to read LINQ statement written in query syntax compared to extension method syntax.\n",
    "\n",
    "However query syntax is very different from the typical syntax seen in C-like languages, so it tends to look out of place. Query syntax also has more limitations than extension method syntax, because it has less methods to choose from, so in more complex scenarios if has to be used in conjunction with extension method syntax. Also because it is more verbose it tends to output more code to achieve the same result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Filter the the person whose name starts with \"J\" and age is greater than 30.\n",
    "\n",
    "var results = \n",
    "    from person in data\n",
    "    where person.FirstName.StartsWith(\"J\") && person.Age > 30\n",
    "    select person;\n",
    "\n",
    "results.DisplayTable();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Select the max age of the person whose name starts with \"J\".\n",
    "// Since there is no Max in query syntax, it leads to a mixed usage of query and method syntax.\n",
    "\n",
    "var maxAge = \n",
    "    (from person in data\n",
    "    where person.FirstName.StartsWith(\"J\")\n",
    "    select person.Age).Max();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// Group the data by the last name of the person,\n",
    "// then select the first person in each group,\n",
    "// then select the first name of the person,\n",
    "// but only of the people whose age is greater than 30.\n",
    "\n",
    "var results = \n",
    "    from person in data\n",
    "    group person by person.LastName into g\n",
    "    select g.First() into firstPerson\n",
    "    where firstPerson.Age > 30\n",
    "    select firstPerson.FirstName;\n",
    "\n",
    "results.DisplayTable();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homemade LINQ\n",
    "\n",
    "In this section we will explore an example how to create extension methods that provide similar functionality to the standard LINQ methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    }
   },
   "outputs": [],
   "source": [
    "// All the methods are generic, so they would work with any type, while providing type safety.\n",
    "// All the methods are extension methods.\n",
    "// Arguments are Funcs, since they are either projection or predicate functions.\n",
    "// Usage of yield is optional, but it does provide better performance.\n",
    "// Methods should return enumerables, where appropriate, so they could be chained.\n",
    "\n",
    "public static IEnumerable<T> Where<T>(\n",
    "    this IEnumerable<T> source, \n",
    "    Func<T, bool> predicate)\n",
    "{\n",
    "    Console.WriteLine(\"Where called\");\n",
    "\n",
    "    foreach (var item in source)\n",
    "    {\n",
    "        if (predicate(item))\n",
    "        {\n",
    "            yield return item;\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "public static T FirstOrDefault<T>(\n",
    "    this IEnumerable<T> source,\n",
    "    Func<T, bool> predicate)\n",
    "{\n",
    "    Console.WriteLine(\"FirstOrDefault called\");\n",
    "\n",
    "    foreach (var item in source)\n",
    "    {\n",
    "        if (predicate(item))\n",
    "        {\n",
    "            return item;\n",
    "        }\n",
    "    }\n",
    "\n",
    "    return default(T);\n",
    "}\n",
    "\n",
    "public static IEnumerable<TTarget> Select<TSource, TTarget>(\n",
    "    this IEnumerable<TSource> source,\n",
    "    Func<TSource, TTarget> projection\n",
    ")\n",
    "{\n",
    "    Console.WriteLine(\"Select called\");\n",
    "\n",
    "    foreach (var item in source)\n",
    "    {\n",
    "        yield return projection(item);\n",
    "    }\n",
    "}\n",
    "\n",
    "// Trying out the above defined methods\n",
    "\n",
    "// Due to the way how notebook works, only the calls in this cell will use these custom methods.\n",
    "// In typical development scenario, these methods could be used by having the `using` directive\n",
    "// with the namespace of class where these methods are defined.\n",
    "\n",
    "data.Where(p => p.FirstName.StartsWith(\"D\")).DisplayTable();\n",
    "\n",
    "// Chaining these methods\n",
    "data.Where(x => x.Age > 30).Select(x => x.FirstName).FirstOrDefault();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises to try out LINQ\n",
    "\n",
    "Using the `data` list defined in the first cell and `Linq` extension methods, try doing the exercises below. \n",
    "\n",
    "*It may not be possible to complete every single exercise with absolutely pure single statement `Linq`.*\n",
    "\n",
    "1. Find the most common last name.\n",
    "2. Find the person whose first name and last name together are the longest.\n",
    "3. Find the oldest person by each last name. \n",
    "4. Check if there is more than 1 person of the same age.\n",
    "5. Check if there is more than 1 person with the same first and last name combination.\n",
    "6. Find the average last name length.\n",
    "7. Return the `{FirstName} {LastName}` of the people with top 3 oldest ages (assume that there can be multiple people of the same age).\n",
    "8. Find the second oldest person.\n",
    "9. Calculate the occurrences of letters in people's names. \n",
    "10. Construct the string of 5 most common first name letters ordered descending."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "polyglot-notebook"
  },
  "polyglot_notebook": {
   "kernelInfo": {
    "defaultKernelName": "csharp",
    "items": [
     {
      "aliases": [],
      "languageName": "csharp",
      "name": "csharp"
     }
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}