Distributed Computing Overview

This is supposed to provide an overview over different techniques. Some of them already supported, or partially supported, or not yet implemented (but all could be done).

• RETURNN multi-GPU training (using Horovod)
• Distributed-TensorFlow
• GShard ...
• Mesh-TensorFlow
• ...