apt-get failures [HTTP proxy support]

[jlkweb12]

I am getting an error executing this command:
/bin/sh -c apt-get update && apt-get -y install sshpass && apt-get clean

It appears to get through most of the second download progress bar and fail. I am behind a corporate proxy so that could be causing an issue.

do you have any ideas how to resolve this issue?

[sclevine]

Have you configured your Docker daemon with your corporate HTTP proxy addresses?

[jlkweb12]

Unfortunately yes with a proxy.conf and in the ENV variables.

[sclevine]

Can you confirm that your Docker daemon is configured properly like this:

opal:spec stephen$ docker run -it cloudfoundry/cflinuxfs2 env
...
HTTP_PROXY=example.com:8080
http_proxy=example.com:8080
HTTPS_PROXY=example.com:8080
https_proxy=example.com:8080
...

[jlkweb12]

Yes,

Even confirmed by running:
sudo systemctl show --property Environment docker

[sclevine]

Does the following succeed?

user@host$ docker run -it cloudfoundry/cflinuxfs2 bash
root@5f7065bf0fd9:/# apt-get update && apt-get -y install sshpass && apt-get clean

Also, please provide the full output of:

docker run -it cloudfoundry/cflinuxfs2 env

with any sensitive info redacted.

[Shuchit]

I am running into the same issue. When I run cflocal from behind a corporate proxy, the "apt" commands inside the docker container fail. This is because the proxy configuration also needs to be set inside the container and there does not seem to be a way to do this.

Typically, you can pass environment variables as part of the "docker run" command. But cflocal hides the docker run command so there needs to be a different mechanism for setting the http_proxy, https_proxy, no_proxy environment variables inside the docker container created by cflocal.

[sclevine]

Can you describe your Docker setup? This seems to work for me, and I believe Docker for Mac/Windows uses a daemon called vpnkit that redirects all container HTTP traffic through an enterprise proxy without those env vars needing to be set.

(FYI: CF Local doesn't actually use the docker CLI at all, it speaks directly to the Docker daemon. The Docker daemon usually handles setting these env vars in all containers if necessary.)

[Shuchit]

Just to summarize, support for proxy is required in two places in docker.

1. docker daemon needs to be able to fetch the correct image and requires the proxy configuration. This is done in the systemctl configuration if you are on Linux and using systemctl for docker.
2. proxy configuration has to be set inside the docker container after it has been created. Typical way to do this is by using "-e" switch when using "docker run" . I am not sure how that would be specified using the docker api that cflocal uses.

I am running this inside a Ubuntu 16 VM. Proxy environment variables are set in bash shell. I have also set the proxy in /etc/systemd/system/docker.service.d/http-proxy.conf. This allows the cflinux2 and cflocal images to be downloaded.

systemctl show --property=Environment docker
shows me the correct values for the proxy variables.

If I create the docker container manually and run env inside, below is the output

$ env
HOSTNAME=f40dfeee1335
TERM=xterm
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arj=01;31:.taz=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lz=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.axa=00;36:.oga=00;36:.spx=00;36:.xspf=00;36:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
HOME=/root
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/env

UPDATE:
For completeness, below is the proxy configuration for the docker-daemon:
$ cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="http_proxy=http://proxy.ykf.mycompany.corp:8080"
Environment="https_proxy=http://proxy.ykf.mycompany.corp:8080"
Environment="no_proxy=.local,localhost,127.0.0.1,169.254/16,.mycompany.corp,.corp.mycompany,172.25/16,192.168.56/24,192.168.99/24,10.7/16,local-dep-dev,.cloud.mycompany"
Environment="HTTP_PROXY=http://proxy.ykf.mycompany.corp:8080"
Environment="HTTPS_PROXY=http://proxy.ykf.mycompany.corp:8080"
Environment="NO_PROXY=.local,localhost,127.0.0.1,169.254/16,.mycompany.corp,.corp.mycompany,172.25/16,192.168.56/24,192.168.99/24,10.7/16,local-dep-dev,.cloud.mycompany"

[sclevine]

I was under the impression that transparent proxying was more widely deployed than it actually is.

Marking this as a bug. This will be easy to fix — expect it to work in the next release.

[Shuchit]

Thanks, much appreciated.
Sadly transparent proxying is being slowly rolled out, but its not going to be fully there for a couple of years.

For completeness, I will add one other change that I needed to do in Docker to get everything working with a proxy and run "apt" commands successfully inside the docker container. By default, the DNS server information that is passed to the docker containers are the Google's free dns servers. These will likely know nothing about the corporate specific proxy servers and the container will fail to resolve the proxy server. Hence, the dns information passed to the containers needs to be overridden as well. I did this by setting DOCKER_OPTS environment variable similar to http_proxy variable. The format is
DOCKER_OPTS=--dns x.x.x.x --dns y.y.y.y --dns 8.8.8.8
For Ubuntu and other Linux systems that use systemd refer to https://docs.docker.com/engine/admin/systemd/#httphttps-proxy

[sclevine]

@jlkweb12 @Shuchit Explicit support for HTTP and HTTPS proxies in containers should no longer be necessary in CF Local as of Docker 17.07. Please see here for instructions. I can confirm this works automatically on macOS (and it's documented to work automatically on macOS here).

Please let me know if upgrading to Docker 17.07 addresses this issue for you.

[sclevine]

@jlkweb12 @Shuchit Let me know if I can close this issue.

[Shuchit]

I tried this for a second time and I was not able to get it to work. I am on Ubuntu16 with docker running as a systemd service. I made sure I am using the latest release of docker 17.12.1-ce. The proxy configuration is passed in through systemd configuration and the docker deamon sees it as an environment variable. I did restart docker service to make sure the configuration is correctly picked up.

It looks like docker daemon can pull down the required images using the proxy. But the container does not get passed the proxy information.

[sclevine]

Thanks for testing this out! Sorry that it's still not working.

It occurs to me now that the logic to parse the ~/.docker/config.json file mentioned in the docs may live in a part of the client that isn't compiled into CF Local v0.18.0. I'll make sure it gets into v0.19.0.

[sclevine]

@Shuchit can you try this release candidate for 0.19.0?

cflocal-0.19.0.rc1-linux.zip

It should respect any HTTP proxy settings in the environment that you're running CF Local in, as long as DOCKER_HOST is set to the default local socket. If you have a custom value for DOCKER_HOST and you want the proxy settings to apply, set CFL_FORCE_PROXY=true.

[sclevine]

(In the official v0.19.0 release, the env var will likely be CFL_USE_PROXY instead.)

[sclevine]

Fixed in v0.19.0.

If you've customized DOCKER_HOST, you must pass CFL_USE_PROXY=true. (This is to prevent local proxy settings from applying to remote Docker daemons.)