Refactor offline bosh release management

[poblin-orange]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

minio s3 full

Describe the solution you'd like
A clear and concise description of what you want to happen.

Automation should only update in minio the required versions

current limitations:
1 - Current implementation eaglerly loads all the successive release.
2- Its does not manage the non bosh.io releases (deploy.sh), except github hosted release (different exposition format exist, git, github, tgz, ...)
3- It does not manage the runtime-config bosh releases
4- it induces excessive internet resource scanning (bosh.io)
5 - no integrity check (sha1)
6- lack of automated cleaning of locally replicated bosh releases following release bumps.

Additional requirements:
7- be able to review downloaded releases and inspect them

proposal 1:

• reduce scope of COA: minio is the reference. runner level tools load minio ?
• provide in paas-templates optionnal pipeline concourse to load the minio s3 with the required source (activable per runner. eg corporate repository, direct internet access ...)

proposal 2:

• a bosh release is denoted by url conforming to the bosh release url format and a sha1 (as per bosh deployment manifest spec)
• the bosh director handles the download of the bosh release, by using the configured http_proxy and no_proxy directives.
• paas-templates brings a reverse caching http proxy (exposed as http://caching-proxy.internal.paas/FQDN/path?query-params such as https://caching-proxy.internal.paas/bosh.io/d/github.com/cloudfoundry/loggregator-agent-release?v=3.9`)
  • It proxifies http requests to upstream FQDN server
  • It caches the 1st successfully downloaded content.
  • See related exchanges with CF/Bosh teams at Support release and stemcell caching by intermediate proxies cloudfoundry/cf-deployment#697 Support serving releases and stemcells over http as to support caching cloudfoundry/bosh-io-web#41
  • Nginx seems to natively support this use-case
    • https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/
    • https://www.google.fr/search?q=nginx+proxy_pass+query+params&ie=utf-8&oe=utf-8&client=firefox-b-e
    • http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
    • 30X redirects from upstream servers (e.g. github or bosh.io) would be rewritten through proxy_redirect directive and returned to bosh director.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[gberche-orange]

@poblin-orange @o-orand I refined the proposal 2 above with a potential implementation with native ngnix support.

[o-orand]

As we changed bosh releases management (we repackage bosh release before usage, see #330 and #339), we implement a pull mechanism close to proposal 1: #355