sidecar: improve upload error handling logic / add retries

[GiedriusS]

Currently we do a best effort clean up if any of the uploads had failed: https://github.com/improbable-eng/thanos/blob/master/pkg/block/block.go#L88: the Upload() method of the Bucket interface tries to do that only once - and if it fails then we try to clean it up. However, it might just happen (like in my case) that uploading one chunk file had succeed, and Thanos failed to upload another one. Then, it tried to delete the files which obviously had not succeeded because there were still issues with the S3 storage. Thus, we should be a bit smarter here and maybe retry the operations more than once in the Upload() method (defined in the bucket configuration) or do that on the outside and try sending a Get() to check if all of the files had been successfully uploaded a few times, and reupload them, if possible.

ACC:

• Add a user configurable option which would make Thanos Sidecar retry uploading (and/or deleting) if it fails. Can either be a per-bucket configuration or a globally configured option.

[bwplotka]

dup of #318

I guess let's keep just one thread for this?