[WIP] Refactor various parts of qiskit using multiple dispatch

[jlapeyre]

This PR refactors some code using multiple dispatch. This is an experiment in applying multiple dispatch in
various situations. The experiments are mostly spread sparsely across the repo. This comment is a WIP, as well.

This PR requires the master branch of plum-dispatch. You can get this by cloning
plum-dispatch and doing, for example in a virtual env, pip install -e ..

Advantages/Motivation

• disentangles forests of isinstance conditionals, making the logic more clear.
• Remove type-checking boilerplate. A method error is thrown if no matching method exists. For example

        if not isinstance(scalar, (int, float, complex, ParameterExpression)):
            raise ValueError('Operators can only be scalar multiplied by float or complex, not '

• Especially useful with mathematical operators. This is by far the heaviest use case in Julia. This is difficult in python because operators must be class methods that privilege one operand. A workaround is to use function calls rather than infix operators.

  • Or, one could have __mul__ and __rmul__ for several classes all call a function (not method) multiple which supports multiple dispatch.

• types representing lists and tuples of homogeneous type simplify logic. An example (not in this PR) is
  https://github.com/jlapeyre/qiskit-terra/blob/253339278eb4e706805ae57653524f67dc8ad6f3/qiskit/opflow/gradients/circuit_gradients/lin_comb.py#L142-L144

  is replaced by something like

def func(params: {tuple, plum.List(tuple)}):

• Currently used mostly within classes in an asymmetric situation. I.e. with gradients the class is doing something to the argument. A bigger advantage is gained when you have an operation between two objects and it is not clear two which class the method should belong. (Explain why: it's not just that you don't have to make this decision; it affords more flexibility)

Multiple-dispatch libraries

Multiple dispatch or multimethods in languages such as Dylan and Common Lisp (as CLOS) is older than the Julia language.
However, much of the motivation and design choices of current Python implementations come from Julia.

• multimethod I have not yet tried this library. It's stated goal is dispatch speed.
• plum-dispatch Probably the most feature-ful
  • Also exposes type conversion and promotion mechanisms modeled on Julia
  • Keyword-only args, separated by *, are supported. They do not enter into dispatch. This is similar to Julia
  • Supports @staticmethod, at least basic.
  • If no matching method is found at dispatch time, a NotFoundLookupError is raised. One could add a feature to plum to print closest matching methods, as Julia does.
• multipledispatch
  • Cannot use default values for args. This should be a possible feature.
  • Lowest time overhead for dispatch in benchmarks (of multipledispatch, plum-dispatch, and fastcore)
• fastcore
  • If there is no matching method, a method may still be called, and also called incorrectly. It would be better to throw some kind of MethodError.
  • Not as featureful as others
  • Small. A single file < 200 LOC.

Design issues

• How to organize documentation. Doc system is not built to accommodate MD

Other considerations

Storing type-like information in various places

In qiskit, we sometimes distinguish "kinds" of objects by attributes; a flag or a string. Or by checking whether a method exists. This design complicates MD and makes it more difficult to apply broadly. I would be a good idea to move these things into the type system. The easiest way is to make another type rather than a use a flag. But, maybe there is also a way to use dispatch with a trait system.

Does a multiple-dispatch system turn Python into Julia? No. The blurred distinction between run time and compile time in Julia offers an advantage that is difficult to reproduce in Python (compilation itself is a big problem in Python). First, let's be more precise by what we mean by multiple dispatch.

Multiple dispatch vs. operator overloading

MD is distinguished from function overloading in that the former is dynamic while the latter is static. Dispatch in the former is done on the dynamic type, in the latter it is done on the statically, lexically, analyzed type. Consider the code

function func(a::Animal)
    make_noise(a)
end

where we have three abstract types with the relation Animal <: Thing, Machine <: Thing, and another type Dog <: Animal. We might have methods

make_noise(m::Machine) = "crash"
make_noise(a::Animal) = "cry"
make_noise(dog::Dog) = "bark"

What happens when I call func(dog) ? In an operator-overloading language I get "cry", the static, lexically, deterimined type of a. In a MD language, dispatch is on the actual type of the object; I get "bark".

Run-time compilation and inference

But, an important advantage is present in both static languages and Julia that is not available in Python and CLOS. In Julia, when func(dog) is first called, the types of all expressions in the body are inferred if possible and code is compiled specially for the inferred types. In particular, although this is "dynamic dispatch", the method make_noise is devirtualized, and inlined if advantageous. This strategy may be pursued to any depth (there are algorithms to determine when it is advantageous not to specialize). One thereby obtains advantages of both static and dynamic languages.

• No diagonal dispatch. Some (all?) MD libraries' docs claim they do no support diagonal dispatch. I think the issue more broadly is that they do not support proper parametric types, in particular type parameters in the parameter lists. plum claims it has a method to get around lack of diagonal dispatch (using Self). But, I don't the problem worked around by Self is stated correctly. It is that @dispatch can't recognize the type of a class inside the definition of the class, because is net yet recognized by python. Self is in some sense a way to defer referring to the class by name.

• No universal, coherent type system.