Remove Chocolate Suggestion Service

[tenzen-y]

/kind discussion

The final commitment of Chocolate was 4 years ago and the community seems to be inactive. So the framework can not work on Python 3.10 since some Python standard libraries are not working on Python 3.10 as shown below:

../../../../../opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/chocolate/conditional/thompson.py:1
/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/chocolate/conditional/thompson.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Sequence

../../../../../opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/chocolate/base.py:1
/opt/hostedtoolcache/Python/3.9.15/x64/lib/python3.9/site-packages/chocolate/base.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
from collections import Mapping, Sequence

IMO, we have the following 3 options to resolve an issue:

1. Remove Chocolate Suggestion Service from katib between 1 and 2 releases.
2. Continue to use Python 3.9 only for the Chocolate Suggestion Service.
3. Fork or hold the Chocolate repository and then maintain the framework by katib community.

I would like to choose the first option since we can provide major Chocolate search algorithms by using other frameworks in the following after we remove that framework:

• Grid Search -> Optuna
• Random Search -> Hyperopt, Goptuna or Optuna
• Bayesian optimization -> Scikit-Optimize
• QuasiRandom -> None (We are not providing this by default)
• MOCMAES -> None (We are not providing this by default)

@kubeflow/wg-automl-leads @anencore94 wdyt?

---

TODO for removing Chocolate Suggestion Service:

• Support grid search algorithm in Optuna Suggestion Service. (Support for grid search algorithm in Optuna Suggestion Service #2060)
• Remove all codes related Chocolate Suggestion Service. (Remove Chocolate Suggestion Service #2071)
• Update kubeflow documentation. ([Katib] Update documentation for suggestion algorithms website#3418)

---

Love this feature? Give it a 👍 We prioritize the features with the most 👍

[johnugeorge]

Agree. Since chocolate is not supported, better to remove over time.

[anencore94]

I agree with you too :)

[andreyvelich]

Yes, I think we should remove Chocolate from our supported suggestions.

Do we have any knowledge which frameworks might support QuasiRandom and MOCMAES ?
We support sobol Quasirandom Sequence from Goptuna, do we know the differences ?

[tenzen-y]

@andreyvelich

Do we have any knowledge which frameworks might support QuasiRandom and MOCMAES ?

I can not find any frameworks to support MOCMAES. For example, optuna provides CmaEsSample, although not to support Multi-Objective.

https://optuna.readthedocs.io/en/stable/reference/samplers/index.html#optuna-samplers

We support sobol Quasirandom Sequence from Goptuna, do we know the differences ?

Each framework seems to be different from the generator for quasi-random numbers.

• Goptuna: original implementation (https://github.com/c-bata/goptuna/blob/2245ddd9e8d1edba750839893c8a618f852bc1cf/sobol/engine.go)

• Optuna: SciPy implementation (https://scipy.github.io/devdocs/reference/stats.qmc.html)
  ref: https://optuna.readthedocs.io/en/latest/reference/samplers/generated/optuna.samplers.QMCSampler.html#optuna-samplers-qmcsampler

  We use the QMC implementations in Scipy. For the details of the QMC algorithm, see the Scipy API references on scipy.stats.qmc.

• Chocolate: ghalton implementation (https://github.com/fmder/ghalton)
  ref: https://chocolate.readthedocs.io/en/latest/api/sample.html#chocolate.QuasiRandom

  Samples the search space using the generalized Halton low-discrepancy sequence. The underlying sequencer is the ghalton package, it must be installed separately.

  ref: https://github.com/fmder/ghalton#generalized-halton-number-generator

  This library allows to generate quasi-random numbers according to the generalized Halton sequence.

/cc @c-bata

[tenzen-y]

I created another issue, #2078, about MOCAMES.
/close

[google-oss-prow]

@tenzen-y: Closing this issue.

In response to this:

I created another issue, #2078, about MOCAMES.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.