Allow for pulling customHyperkubeImage from private repository

[jsturtevant]

Is this a request for help?:

no

Is this an ISSUE or FEATURE REQUEST? (choose one):

feature request

What version of acs-engine?:

any

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
kuberentes

What happened:
Would like to be able to pull customHyperkubeImage as describe in https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/k8s-developers.md from a private Azure Container Registry.

Would it be possible to pass Service Principle credentials or use MSI to pull the image? What would be the effort for making this possible?

Looks like this happens in https://github.com/Azure/acs-engine/blob/8b616722debcddaf0d0d9d2baa605bb7aa77eddb/parts/k8s/kubernetesinstalls.sh#L178:1

Anything else we need to know:
cc: @PatrickLang and @ritazh.

[PatrickLang]

I don't think MSI makes sense at this point. Each node in a scale set and the masters will need to be able to pull images right away, so I think that means the login needs to be in the ARM template. If you create a SP and give it access only to the ACR instance - you use the clientid as username, and password for docker login. I think adding user/pass support makes sense as a starting point.

[PatrickLang]

Reference on ACR login: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication

[khenidak]

What is the use case for this?

[PatrickLang]

Deploying windows+Linux nodes from the same commit during k8s development cycles. A build server pushes the hyperkube image to ACR as a step in that build.

[andyzhangx]

If you could grant your AKS Service Principle with that ACR access, then it would work.
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-aks#grant-aks-access-to-acr

[andyzhangx]

o, i see, I got the point, it's not app image...
usually I put the hyperkube image on dockerhub since acr does not provide anonymous read access now. ACR team has plan to suppor that in the future:
https://feedback.azure.com/forums/903958-azure-container-registry/suggestions/32517127-enable-anonymous-access-to-registries

[jchauncey]

Definitely think this is an issue we should consider. When building private hyperkube images from our CI system for testing I would rather not put those in a public repo. Instead it would be great to place them in ACR which is private only. Cursory thoughts on how we might possibly do this is to just have an extension do a docker login with some provided credentials. Thoughts?