In order to compare performance of Vowpal Wabbit across versions a consistent set of benchmarks should be used, this repo aims to collect these benchmarks and make them easily reproducible and visualizable. Benchmarks are command line based so they are simple to specify. Also because VW's primary interface is command line and so this is the only way to be able to run a set of benchmarks over many versions.