Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add function to parse quantity #37

Merged
merged 13 commits into from
Jul 8, 2022
Merged
99 changes: 99 additions & 0 deletions docs/utils.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Utils

## Convert quantity string to decimal

K8s converts user input
[quantities](https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/)
to "canonical form":

> Before serializing, Quantity will be put in "canonical form". This means that
> Exponent/suffix will be adjusted up or down (with a corresponding increase or
> decrease in Mantissa) such that: a. No precision is lost b. No fractional
> digits will be emitted c. The exponent (or suffix) is as large as possible.
> The sign will be omitted unless the number is negative.
>
> Examples: 1.5 will be serialized as "1500m" 1.5Gi will be serialized as "1536Mi"

Additional examples:

| User input | K8s representation |
|----------------------------------|-------------------------------|
| `{"memory": "0.9Gi"}` | `{"memory": "966367641600m"}` |
| `{"cpu": "0.30000000000000004"}` | `{"cpu": "301m"}` |

You may need to compare different quantities when interacting with K8s.

## Interface

::: lightkube.utils.quantity.parse_quantity
:docstring:

::: lightkube.utils.quantity.equals_canonically
:docstring:

## Examples

After patching a statefulset's resource limits you may want to compare
user's input to the statefulset's template to the active podspec:

```python
>>> from lightkube import Client
>>> from lightkube.models.apps_v1 import StatefulSetSpec
>>> from lightkube.models.core_v1 import (
... Container,
... PodSpec,
... PodTemplateSpec,
... ResourceRequirements,
... )
>>> from lightkube.resources.apps_v1 import StatefulSet
>>> from lightkube.resources.core_v1 import Pod
>>> from lightkube.types import PatchType
>>>
>>> resource_reqs = ResourceRequirements(
... limits={"cpu": "0.8", "memory": "0.9Gi"},
... requests={"cpu": "0.4", "memory": "0.5Gi"},
... )
>>>
>>> client = Client()
>>> statefulset = client.get(StatefulSet, name="prom", namespace="mdl")
>>>
>>> delta = StatefulSet(
... spec=StatefulSetSpec(
... selector=statefulset.spec.selector,
... serviceName=statefulset.spec.serviceName,
... template=PodTemplateSpec(
... spec=PodSpec(
... containers=[Container(name="prometheus", resources=resource_reqs)]
... )
... ),
... )
... )
>>>
>>> client.patch(
... StatefulSet,
... "prom",
... delta,
... namespace="mdl",
... patch_type=PatchType.APPLY,
... field_manager="just me",
... )
>>>
>>> client.get(StatefulSet, name="prom", namespace="mdl").spec.template.spec.containers[1].resources
ResourceRequirements(limits={'cpu': '800m', 'memory': '966367641600m'}, requests={'cpu': '400m', 'memory': '512Mi'})
>>>
>>> pod = client.get(Pod, name="prom-0", namespace="mdl")
>>> pod.spec.containers[1].resources
ResourceRequirements(limits={'cpu': '800m', 'memory': '966367641600m'}, requests={'cpu': '400m', 'memory': '512Mi'})
>>>
>>> from lightkube.utils.quantity import parse_quantity
>>> parse_quantity(pod.spec.containers[1].resources.requests["memory"])
Decimal('536870912.000')
>>> parse_quantity(pod.spec.containers[1].resources.requests["memory"]) == parse_quantity(resource_reqs.requests["memory"])
True
>>>
>>> from lightkube.utils.quantity import equals_canonically
>>> equals_canonically(pod.spec.containers[1].resources.limits, resource_reqs.limits)
True
>>> equals_canonically(pod.spec.containers[1].resources, resource_reqs)
True
```
9 changes: 8 additions & 1 deletion lightkube/core/internal_models.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import sys

try:
from ..models import meta_v1, autoscaling_v1
from ..models import meta_v1, autoscaling_v1, core_v1
gtsystem marked this conversation as resolved.
Show resolved Hide resolved
except:
if sys.modules["__main__"].__package__ != 'mkdocs': # we ignore this import error during documentation generation
raise
Expand All @@ -19,3 +19,10 @@ class Scale:

autoscaling_v1 = mock.Mock()
autoscaling_v1.Scale = Scale


class ResourceRequirements:
pass

core_v1 = mock.Mock()
core_v1.ResourceRequirements = ResourceRequirements
142 changes: 142 additions & 0 deletions lightkube/utils/quantity.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
import decimal
gtsystem marked this conversation as resolved.
Show resolved Hide resolved
import re
from typing import Optional, overload

from ..core.internal_models import core_v1

ResourceRequirements = core_v1.ResourceRequirements

MULTIPLIERS = {
# Bytes
"m": (10, -3), # 1000^(-1) (=0.001)
"": (10, 0), # 1000^0 (=1)
"k": (10, 3), # 1000^1
"M": (10, 6), # 1000^2
"G": (10, 9), # 1000^3
"T": (10, 12), # 1000^4
"P": (10, 15), # 1000^5
"E": (10, 18), # 1000^6
"Z": (10, 21), # 1000^7
"Y": (10, 24), # 1000^8

# Bibytes
"Ki": (1024, 1), # 2^10
"Mi": (1024, 2), # 2^20
"Gi": (1024, 3), # 2^30
"Ti": (1024, 4), # 2^40
"Pi": (1024, 5), # 2^50
"Ei": (1024, 6), # 2^60
"Zi": (1024, 7), # 2^70
"Yi": (1024, 8), # 2^80
}

# Pre-calculate multipliers and store as decimals.
MULTIPLIERS = {k: decimal.Decimal(v[0])**v[1] for k, v in MULTIPLIERS.items()}


def parse_quantity(quantity: Optional[str]) -> Optional[decimal.Decimal]:
"""Parse a quantity string into a bare (suffix-less) decimal.

K8s converts user input to a canonical representation. For example, "0.9Gi" would be converted
to "966367641600m".
This function can be useful for comparing user input to actual values, for example comparing
resource limits between a statefulset's template
(statefulset.spec.template.spec.containers[i].resources) and a scheduled pod
(pod.spec.containers[i].resources) after patching the statefulset.

**Parameters**

* **quantity** `str` - An str representing a K8s quantity (e.g. "1Gi" or "1G"), per
https://kubernetes.io/docs/reference/kubernetes-api/common-definitions/quantity/.

**returns** An instance of `decimal.Decimal` representing the quantity as a bare decimal.
"""
if quantity is None:
# This is useful for comparing e.g. ResourceRequirements.limits.get("cpu"), which can be
# None.
return None

pat = re.compile(r"([+-]?\d+(?:[.]\d*)?(?:e[+-]?\d+)?|[.]\d+(?:e[+-]?\d+)?)(.*)")
match = pat.match(quantity)

if not match:
raise ValueError("Invalid quantity string: '{}'".format(quantity))

try:
value = decimal.Decimal(match.group(1))
except ArithmeticError as e:
raise ValueError("Invalid numerical value") from e

unit = match.group(2)

try:
multiplier = MULTIPLIERS[unit]
except KeyError:
raise ValueError("Invalid unit suffix: {}".format(unit))

try:
as_decimal = value * multiplier
return as_decimal.quantize(decimal.Decimal("0.001"), rounding=decimal.ROUND_UP)
except ArithmeticError as e:
raise ValueError("Invalid numerical value") from e


def _equals_canonically(first_dict: Optional[dict], second_dict: Optional[dict]) -> bool:
"""Compare resource dicts such as 'limits' or 'requests'."""
if first_dict == second_dict:
# This covers two cases: (1) both args are None; (2) both args are identical dicts.
return True
if first_dict and second_dict:
if first_dict.keys() != second_dict.keys():
# The dicts have different keys, so they cannot possibly be equal
return False
return all(
parse_quantity(first_dict[k]) == parse_quantity(second_dict[k])
for k in first_dict.keys()
)
if not first_dict and not second_dict:
# This covers cases such as first=None and second={}
return True
return False


@overload
def equals_canonically(first: ResourceRequirements, second: ResourceRequirements) -> bool:
...


@overload
def equals_canonically(first: Optional[dict], second: Optional[dict]) -> bool:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see sufficient benefits of supporting dicts. Users can always convert dicts to ResourceRequirements (ResourceRequirements.from_dict). I'll suggest to drop this for now, so that we can have a simpler implementation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dict here is not an alternative representation of
ResourceRequirements, but of ResourceRequirements.limits and ResourceRequirements.requests, i.e. one level down.

The tests demonstrate this. With the overload we're able to do both:

equals_canonically({"cpu": "0.6"}, {"cpu": "600m"})

and

equals_canonically(
    ResourceRequirements(limits={"cpu": "0.6"}), 
    ResourceRequirements(limits={"cpu": "600m"})
)

If feasible, we could change ResourceRequirements to use TypedDict

+class ResourceSpecDict(TypedDict, total=False):
+    cpu: Optional[str]
+    memory: Optional[str]

@dataclass
class ResourceRequirements(DataclassDictMixIn):
-    limits: 'dict' = None
-    requests: 'dict' = None
+    limits: ResourceSpecDict = None
+    requests: ResourceSpecDict = None

In which case the overload would become

@overload
def equals_canonically(first: Optional[ResourceSpecDict], second: Optional[ResourceSpecDict]) -> bool:

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, I was confused by the function description "Compare two resource requirements for numerical equality." I assumed that the only acceptable type is ResourceRequirements or the equivalent dict. Maybe we can clarify further?

No need to introduce the TypedDict as anyway all the keys are optional and we can have further keys.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in next commit.

...


def equals_canonically(first, second):
"""Compare two resource requirements for numerical equality.

Both arguments must be of the same type and can be either:
- `ResourceRequirements`; or
- Optional[dict], representing the "limits" or the "requests" portion of `ResourceRequirements`.

>>> equals_canonically({"cpu": "0.6"}, {"cpu": "600m"})
True
>>> equals_canonically(ResourceRequirements(limits={"cpu": "0.6"}), ResourceRequirements(limits={"cpu": "600m"}))
True

**Parameters**

* **first** `ResourceRequirements` or `dict` - The first item to compare.
* **second** `ResourceRequirements` or `dict` - The second item to compare.

**returns** True, if both arguments are numerically equal; False otherwise.
"""
if isinstance(first, (dict, type(None))) and isinstance(second, (dict, type(None))):
# Args are 'limits' or 'requests' dicts
return _equals_canonically(first, second)
elif isinstance(first, ResourceRequirements) and isinstance(second, ResourceRequirements):
# Args are ResourceRequirements, which may contain 'limits' and 'requests' dicts
ks = ("limits", "requests")
return all(_equals_canonically(getattr(first, k), getattr(second, k)) for k in ks)
else:
raise TypeError("unsupported operand type(s) for canonical comparison: '{}' and '{}'".format(
first.__class__.__name__, second.__class__.__name__,
))
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
author_email='gtsystem@gmail.com',
license='MIT',
url='https://github.com/gtsystem/lightkube',
packages=['lightkube', 'lightkube.config', 'lightkube.core'],
packages=['lightkube', 'lightkube.config', 'lightkube.core', 'lightkube.utils'],
package_data={'lightkube': ['py.typed']},
install_requires=[
'lightkube-models >= 1.15.12.0',
Expand Down
Loading