[v1.7.0-rc.1] MySQL binary logs grow fast and indefinitely in size

[TobiasGoerke]

Having updated to Kubeflow 1.7, our mysql database ran out of storage within less than a day (20GB). I've doubled its storage just to see it run full again even sooner.

Upon investigation, it became obvious that mysql's binary logs, which are stored on the volume under /var/lib/mysql are to blame for this (see screenshot).

I'm sure many Kubeflow users will have the same issue after updating. I'm suggesting to completely disabling binlogs using '--disable-log-bin'. Setting 'binlog-expire-logs' to 10 days, as set in pipelines didn't work for me as the storage ran full much sooner than that. Also, the patch was introduced in December and isn't included in the current RC.

[TobiasGoerke]

Fixed with #8941. Thanks!