Local Changes Requires Docker Hub Auth

[mehmetbebek]

I just want to make faster local development. At our company we have different cicd pipeline so I shouldn't send new images a hub.
If there is a small change I am following next commands to achieve changes. It takes time.

cd Mgnfcnt-Project;
rm -rf ./target;
mvn clean install -DskipTests;
docker stop Mgnfcnt-Project && docker rm Mgnfcnt-Project && docker rmi mgnfcnt_Mgnfcnt-Project;
docker-compose up -d Mgnfcnt-Project;

I dont know whether JIB is right solution or not for my problem but using as following:

      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>2.1.0</version>
        <configuration>
          <from>
            <image>mgnfcnt_Mgnfcnt-Project:latest</image>
          </from>
          <to>
            <image>mgnfcnt_Mgnfcnt-Project</image>
          </to>
        </configuration>
      </plugin>

But still wants to reach docker hub as gives following error:

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.1.0:build (default-cli) on project Mgnfcnt-Project: Build image failed, perhaps you should make sure your credentials for 'registry-1.docker.io/library/mgnfcnt_Mgnfcnt-Project' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help

Thanks in advance

[chanseokoh]

Hi @mgnfcnt,

For the base image, you are using a tag :latest, which can refer to different images at different times. Jib ensures that you are using the up-to-date image by asking Docker Hub if Jib needs to fetch a newer image. For up-to-date checking, Docker Hub just returns a small JSON; this checking is very light-weight. If :latest didn't change and is already cached, Jib won't download the image again. It is not that Jib is trying to push something.

Moreover, do note that, even if :latest is updated to point to a newer base image, Jib will still not send layers for your application to Docker Hub, because there was no change to your application; in this case, Jib only needs to send the updated base image (only if the base image is missing in your target repository).

Consider using a digest (e.g., mgnfcnt_Mgnfcnt-Project@sha256:...) for better reproducibility. In this case, Jib doesn't need to reach out to Docker Hub, because a digest referred to by a digest always refers to the same image.

[chanseokoh]

Oh, I just noticed your target repository is also mgnfcnt_Mgnfcnt-Project and you are explicitly instructing Jib to push an image to a remote registry (Docker Hub). Therefore, in addition to what I explained regarding the base image, Jib does need to contact Docker Hub to check if your image (actually, layers comprising your image) is already uploaded. But as I said, if the layers are already uploaded, nothing will be uploaded.

[mehmetbebek]

@chanseokoh Thanks for your answers, because of these are local images, I am producing them via docker-compose up --build command.

Because of this when I run docker images --digests command it doesn't give any sha256 code
it is just none

[chanseokoh]

And one tip using docker-compose. It is no wonder the following is slow.

docker stop Mgnfcnt-Project && docker rm Mgnfcnt-Project && docker rmi mgnfcnt_Mgnfcnt-Project;
docker-compose up -d Mgnfcnt-Project;

This will be far faster:

docker stop Mgnfcnt-Project && docker rm Mgnfcnt-Project && docker pull mgnfcnt_Mgnfcnt-Project;
docker-compose up -d Mgnfcnt-Project;

And I don't really get why you do mvn clean install during development. If all you want is a fast development cycle using Jib and docker-comose

cd Mgnfcnt-Project;
mvn jib:dockerBuild # or if necessary, mvn compile jib:dockerBuild
docker stop Mgnfcnt-Project && docker rm Mgnfcnt-Project
docker-compose up -d Mgnfcnt-Project;

[mehmetbebek]

Why I need to make clean install is creating jar file under target, This is my Dockerfile.local used by docker-compose.yml

FROM adoptopenjdk/openjdk8:jdk8u242-b08-alpine

RUN apk --update add bash curl

# Create java user
ARG user=java
ARG uid=1000
ARG home=/home/java

RUN adduser -S ${user} -u ${uid} -s /bin/sh && \
	mkdir -p /opt/java


COPY ./target/Mgnfcnt-Project*.jar /opt/java/Mgnfcnt-Project.jar

EXPOSE 8090
EXPOSE 5005

USER ${user}
WORKDIR ${home}


CMD ["java", "-jar","-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005", "/opt/java/Mgnfcnt-Project.jar", "--server.port=8090"]

[chanseokoh]

Oh, you are not using Jib yet. If you use Jib, it would just be

docker stop Mgnfcnt-Project && docker rm Mgnfcnt-Project
cd Mgnfcnt-Project
mvn jib:dockerBuild # or if necessary, mvn compile jib:dockerBuild
docker-compose up -d Mgnfcnt-Project

and this will be super fast. Try yourself. And if your base image is very large, jib:build (jib in Gradle) can actually be faster than jib:dockerBuild (jibDockerBuild in Gradle): https://stackoverflow.com/a/60872181/1701388

Lastly, it's probably not right to use same repository mgnfcnt_Mgnfcnt-Project for the base image and the target image. It doesn't make much sense to me in this case.

[mehmetbebek]

@chanseokoh The problem is I dont have sha codes for local images so it is trying to retrieve from docker hub but my images not located any hub server.

[chanseokoh]

@mgnfcnt I am not sure if you are following all of these. Or maybe we are not on the same page. In this case, I think

• You don't have to use a SHA for the base image. If you want, you can use it though.
• You don't have to push your images to Docker Hub. If you want, you can push it though.

I thought your main goal is to achieve fast local development? For that matter, you can just apply the Jib plugin and do what I said in #2383 (comment)

Sorry, I have to go offline for some time for now. However, I am not sure what exactly the issue you are facing is or what the ultimate goal you are trying to achieve is.

[mehmetbebek]

@chanseokoh Let me start from initial step:

I have several spring boot projects and they all run on docker. I have a docker-compose.yml file as:

...
 mgnfcnt-project:
    container_name: mgnfcnt-project
    build:
      context: Mgnfcnt-Project
      dockerfile: ./Dockerfile.local
    ports:
      - "8095:8090"
      - "5020:5005"
...

And this is my Dockerfile.local

FROM adoptopenjdk/openjdk8:jdk8u242-b08-alpine

RUN apk --update add bash curl

# Create java user
ARG user=java
ARG uid=1000
ARG home=/home/java

RUN adduser -S ${user} -u ${uid} -s /bin/sh && \
	mkdir -p /opt/java


COPY ./target/mgnfcnt-project*.jar /opt/java/mgnfcnt-project.jar

EXPOSE 8090
EXPOSE 5005

USER ${user}
WORKDIR ${home}


CMD ["java", "-jar","-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005", "/opt/java/mgnfcnt-project.jar", "--server.port=8090"]

While development if I make any code change, I am doing following ones in order to reflect the changes:

cd mgnfcnt-project;
rm -rf ./target;
mvn clean install -DskipTests;
docker stop mgnfcnt-project && docker rm mgnfcnt-project && docker rmi mgnfcnt_mgnfcnt-project;
docker-compose up -d mgnfcnt-project;

It takes for a while and making this each code change is waste of time. So I thought jib can be better solution in order to achive, reflecting each change to docker container faster.

In order to achive this, I just added following to pom.xml

      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>2.1.0</version>
        <configuration>
          <from>
            <image>mgnfcnt_mgnfcnt-project</image>
          </from>
          <to>
            <image>mgnfcnt_mgnfcnt-project</image>
          </to>
        </configuration>
      </plugin>

Also tried to add sha key.but because this image doesn't exists at docker hub and won't pushed to there it doesn't have any sha code.

When I use docker images --digests my image's digest code is "none" as expected.

Only docker inspect mgnfcnt_mgnfcnt-project gives sha codes also tried it. But all gives following error:

Base image mgnfcnt_mgnfcnt-project' does not use a specific image digest - build may not be reproducible

and

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.1.0:dockerBuild (default-cli) on project mgnfcnt-project: Build to Docker daemon failed, perhaps you should make sure your credentials for 'registry-1.docker.io/library/mgnfcnt_mgnfcnt-project' are set up correctly. See https://github.com/GoogleContainerTools/jib/blob/master/docs/faq.md#what-should-i-do-when-the-registry-responds-with-unauthorized for help

[chanseokoh]

Thanks for the update. Here is basically what you need to do

1. Remove the following build: part, because you'll make Jib build the mgnfcnt-project image, not using Docker with a Dockerfile. (You may delete the Dockerfile.local file if you don't need it.)

     mgnfcnt-project:
         container_name: mgnfcnt-project
   +     image: mgnfcnt_mgnfcnt-project
   -     build:
   -       context: Mgnfcnt-Project
   -       dockerfile: ./Dockerfile.local
         ports:
           - "8095:8090"
           - "5020:5005"

2. Configure Jib correctly. For the base image, you should use adoptopenjdk/openjdk8:jdk8u242-b08-alpine. And adding a new user may not be necessary. (But if you really want to add a new user, there's a way to do this in Jib.) Something like this (untested):

      <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>jib-maven-plugin</artifactId>
        <version>2.1.0</version>
        <configuration>
          <from>
            <image>adoptopenjdk/openjdk8:jdk8u242-b08-alpine</image>
          </from>
          <to>
            <image>mgnfcnt_mgnfcnt-project</image>
          </to>
          <container>
            <ports><port>8090</port><port>5005</port></ports>
            <user>1000</user>
            <workingDirectory>/home/java</workingDirectory>
            <jvmFlags>
              <arg>-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005</arg>
            </jvmFlags>
            <args>
              <arg>--server.port=8090</arg>
            <args>
          </container>
        </configuration>
      </plugin>

3. Whenever you make a change to your app, you would do

   docker stop Mgnfcnt-Project && docker rm Mgnfcnt-Project
   cd Mgnfcnt-Project
   mvn compile jib:dockerBuild
   docker-compose up -d Mgnfcnt-Project
   

   But as I said, sometimes it can be faster to push to a remote registry with jib:build than pushing to your local Docker daemon with jib:dockerBuild if your image is huge (and if you have decent network bandwidth): https://stackoverflow.com/a/60872181/1701388

[mehmetbebek]

@chanseokoh Thanks for your patience and kindly answers. Now I am trying to do above steps you mention. But it took 20 mins to 55% for mvn compile jib:dockerBuild

[INFO] Executing tasks:
[INFO] [================= ] 55.0% complete
[INFO] > pulling base image layer sha256:e06a83dd0d815...

And it gave timeout several times:

[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.1.0:dockerBuild (default-cli) on project Mgnfcnt-Project: Read timed out -> [Help 1]

I hope this is related with my poor network connectivity but I don't know what is trying to be downloaded because I already have adoptopenjdk/openjdk8 image. And I veried it with docker images command.

adoptopenjdk/openjdk8 jdk8u242-b08-alpine 5661aecdb86f 7 days ago 227MB

My next question is, as I already mention before, there are several other services and I run their container simple docker-compose up command.

If I remove following and add image:

    build:
      context: Mgnfcnt-Project
      dockerfile: ./Dockerfile.local

There wont be image at initial run, and I need to run all these service one by one, right?

Is there any better way to achieve this?

[mehmetbebek]

@chanseokoh And also I couldn't find RUN command's equivalence in container section:

https://github.com/GoogleContainerTools/jib/tree/master/jib-maven-plugin#container-object

I need to run these kind of commands for some services also

RUN apk update
RUN apk --no-cache add ca-certificates openssl wget
RUN update-ca-certificates
RUN wget -qO /app/newrelic.jar https://repo1.maven.org/maven2/com/newrelic/agent/java/newrelic-agent/4.7.0/newrelic-agent-4.7.0.jar
RUN touch /app/newrelic.yml

[chanseokoh]

I already have adoptopenjdk/openjdk8 image. And I veried it with docker images command.

Note that Jib doesn't require Docker and doesn't make use of it. Therefore, Jib needs to download and cache the base image for the first time. If your network is slow, I recommend you run Jib in a separate terminal, leave it open, and forget it. Once Jib cached the base image, the next build will be super fast.

Another workaround is to instruct Jib to get the base image stored in your Docker daemon with <from><image>docker://adoptopenjdk/openjdk8:...</image></from>. But I recommend you eventually make Jib download the base image into its own cache and use it instead of saying docker://..., as using a local Docker daemon engine image may incur some overhead. But this will certainly work.

My next question is, as I already mention before, there are several other services

Sorry, it was a long time ago that I used docker-compose, and I forgot that you need image: in place of build: So, I was wrong about this:

And I also think the <to> image should be mgnfcnt-project instead of mgnfcnt_mgnfcnt-project, because in your docker-compose, you are referring it to mgnfcnt-project

Let's assume that your docker-compose.yml looks like this:

mgnfcnt-project:
    container_name: mgnfcnt-project
    build:
        context: Mgnfcnt-Project
        dockerfile: ./Dockerfile.local
some-service:
    container_name: some-service
    build:
        context: some-service-project-root
        dockerfile: ./Dockerfile.local
another-service:
    container_name: another-service
    build:
        context: another-service-project-root
        dockerfile: ./Dockerfile.local

You don't have to convert every service to use Jib right now. So, I would go first with mgnfcnt-project and continue to build other services using Docker. Now, instead of using build:, you will specify image:.

mgnfcnt-project:
    container_name: mgnfcnt-project
    image: mgnfcnt_mgnfcnt-project  # should match <image><to>
some-service:
    container_name: some-service
    build:
        context: some-service-project-root
        dockerfile: ./Dockerfile.local
another-service:
    container_name: another-service
    build:
        context: another-service-project-root
        dockerfile: ./Dockerfile.local

Then, when doing docker-compose up, it will use the mgnfcnt_mgnfcnt-project image stored in your Docker daemon. Therefore, you just need to run mvn jib:dockerBuild before docker-compose to keep the image up-to-date.

Does this make sense?

[chanseokoh]

And it gave timeout several times:

You may try increasing timeouts with mvn -Djib.httpTimeout=7200000 jib:dockerBuild (or -Djib.httpTimeout=0 for infinity). But this doesn't always work, if your OS imposes a timeout at a lower network level.

Regarding RUN, it's another topic requiring more explanations. The short answer is that you cannot execute commands like in Dockerfile. However, Jib provides a way to put any arbitrary files into an image. But let's first focus on making Jib work first. I can explain this one later.

[mehmetbebek]

@chanseokoh it seems building but changes are not reflecting in image. It prints older value always.

when I chack docker images; there is no newer one

[chanseokoh]

There must be something you are mssing. Did you do mvn compile jib:dockerBuild after changes? Do you see the same result when doing docker run? Could it be that there is an old container running?

[mehmetbebek]

@chanseokoh Environment variables aren't being reflected:
I have added following to pom.xml plugin:

           <environment>
              <http_proxy>http proxy</http_proxy>
              <https_proxy>https proxy</https_proxy>
            </environment>

Because of these environment variables not being setted, it gives null pointer exception:

            val proxyVar = env!!.getProperty("http_proxy")
            var proxyHost = proxyVar!!.substring(0, proxyVar.lastIndexOf(':'))
            var proxyPort = proxyVar.substring(proxyVar.lastIndexOf(':') + 1)

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'uaaRestTemplate' defined in class path resource [/UaaToken.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.client.RestTemplate]: Factory method 'uaaRestTemplate' threw exception; nested exception is java.lang.StringIndexOutOfBoundsException: String index out of range: -1

[chanseokoh]

It is not a null pointer exception but an index out of bounds. It means the value doesn't contain the colon (:). Can you print out the actual value?

[mehmetbebek]

@chanseokoh I have removed environment variables from plugin because they already exist at docker-compose.yml file:

    environment:
      - http_proxy=http://proxy.magnificentcompany.com:80
      - https_proxy=http://proxy.magnificentcompany.com:80
      - no_proxy=localhost,127.0.0.1,*.magnificentcompany.com,mgnfcnt-project

Container running but I have auth issues:

{"name":"RequestError","message":"Error: socket hang up","cause":{"code":"ECONNRESET"},"error":{"code":"ECONNRESET"},"options":{"headers":{"authorization":"Bearer ..."},"url":"http://localhost:8095/mgnfcnt-project/api/check?","method":"POST","resolveWithFullResponse":true,"json":true,"body":{},"simple":true,"transform2xxOnly":false}}

[mehmetbebek]

@chanseokoh What is port mapping equivalence at jib?

this is from docker-compose.yml

    ports:
      - "8095:8090"
      - "5020:5005"

[mehmetbebek]

And also, somehow spring boot starting from 8080 port, normally it should be 8090

2020-04-04 21:17:47.316 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '/mgnfcnt-project'

[mehmetbebek]

@chanseokoh I made a temporary solution and added server.port to application.properties of the spring boot project,

Changes are being reflected. I just have one concern; it takes 44 seconds for mvn compile jib:dockerBuild;

Is it normal?

[chanseokoh]

And also, somehow spring boot starting from 8080 port, normally it should be 8090

The default port of a Spring Boot app is 8080. You can change it in many different ways. One option is to pass --server.port=8090 to the app.

    <container>
         ...
        <args>
            <arg>--server.port=8090</arg>
        <args>
    </container>

@chanseokoh What is port mapping equivalence at jib?

this is from docker-compose.yml

    ports:
      - "8095:8090"
      - "5020:5005"

This is the docker-compose runtime port mapping. It doesn't have anything to do with building a Docker image. You just need to make your Spring Boot app listen on 8090. The docker-compose setting just means that it will forward connection on your host's port 8095 to port 8090 inside your container.

Container running but I have auth issues:

{"name":"RequestError","message":"Error: socket hang up","cause":{"code":"ECONNRESET"},"error":{"code":"ECONNRESET"},"options":{"headers":{"authorization":"Bearer ..."},"url":"http://localhost:8095/mgnfcnt-project/api/check?","method":"POST","resolveWithFullResponse":true,"json":true,"body":{},"simple":true,"transform2xxOnly":false}}

I'm sorry, but I just cannot resolve all of your application and proxy issues. What is certain is that, if there's something not working around some environment variables, then it must be some misconfiguration on your side. At least I can say Jib will inherit all the environment variables from the base image and add environment variables configured in pom.xml. An example is

    <container>
         ...
        <environment>
            <PATH>/usr/bin</PATH>
            <JAVA_HOME>/usr/lib/jvm/openjdk-11</JAVA_HOME>
        </environment>
    </container>

[chanseokoh]

Is it normal?

No, it is not normal.

[chanseokoh]

What's the result of the following? I mean the time for running each command.

$ time mvn compile
# make a simple change to some .java file
$ time mvn compile
$ time mvn jib:dockerBuild
$ time mvn jib:dockerBuild  # second run without making any change

[chanseokoh]

And are you still using docker://... for the base image or not?

[mehmetbebek]

First of all, thank you very much for all your efforts. I feel I am misunderstood about something. I have no intention of configuring you my entire system. Here I read all the documentation before asking you, I am looking at all previously opened issues related to specific problems. The main problem here is that jib does not have good documentation. Here I write all the configurations so that others with the same error can solve their problems in a short time.

[chanseokoh]

No worries. Thanks for the feedback. If you had any trouble getting information from the Maven plugin doc (or any other docs), please do point out and we will improve it.

[mehmetbebek]

time mvn compile

real 0m29.532s
user 0m0.214s
sys 0m0.676s

time mvn compile -- after a small change

real 0m29.455s
user 0m0.167s
sys 0m0.801s

time mvn jib:dockerBuild

real 0m28.201s
user 0m0.060s
sys 0m0.783s

time mvn jib:dockerBuild --second try

real 0m27.597s
user 0m0.152s
sys 0m0.677s

[mehmetbebek]

Using following one, also tried docker one with/out sha, and also tried to add sha for following

          <from>
            <image>adoptopenjdk/openjdk8:jdk8u242-b08-alpine</image>
          </from>

[chanseokoh]

All of your commands are taking ~29 seconds, which seems very unusual. Note that mvn compile doesn't involve Jib at all, and even the first mvn compile is taking a lot of time. I think there is something particular about your Maven setup and/or your environment that is slowing you down. So I think it is not Jib but something else accountable for the long Maven runs. For example, on my machine with a simple Spring Boot app, consecutive mvn compile runs (with no changes and no re-compilation) take less than 2 seconds. What do you think?

[mehmetbebek]

@chanseokoh I will investigate the issue apart from this. I just wanted to know what is the ideal execution time of jib command. I am glad jib is working on my machine finally. I think we can close the issue. Also I am looking forward to hear how to add "RUN" command to maven jib plugin. It is not urgent but I will need it at future.

Thank you very much

[chanseokoh]

On a fast network, ideally consecutive jib:build (i.e., remote registry push) runs without any changes should take a few seconds. If you make a change only to your application, normally it should still take a few or several seconds (excluding the time to rebuild your classes with mvn compile).

jib:dockerBuild should usually be comparably fast. It can be faster than jib:build if your image is reasonably small and your machine is good enough; it can be slower if your image is huge.

I will update here about "RUN" later.

[chanseokoh]

Updates after chatting privately with @mgnfcnt:

One reason Jib was slow was due to the slow network. It was taking >14 seconds to download an image manifest (which is a small JSON).

TIMED   Pulling base image manifest : 14403.0 ms

Using a SHA as explained in #2383 (comment), is one way to avoid checking the manifest online.

Another option is force offline by mvn --offline .... This makes Jib use the cached manifest (not in the local Docker daemon cache but Jib's own cache). Note that the cached manifest may become out of sync with the tag on Docker Hub.

TIMED   Loading to Docker daemon : 10644.0 ms

And loading an image to Docker was taking 10 seconds, which may not be so bad depending on the size of the image. I think the machine is not so powerful.

Note that, as explained before, pushing directly to a registry with jib:build and then doing docker pull can be faster than jib:dockerBuild, if you are on a fast network. Registry API is much more capable than Docker Engine API, so Jib will be able to upload only those layers that do need to be uploaded. docker pull also only pulls missing layers. But if you are on a slow network, one elaborate setup to speed things up is to run a local Docker registry; you do jib:build to push to the local registry (i.e., not the local Docker daemon cache), and then do docker pull. It's very easy to run a local Docker registry, but of course, it's indeed another burden to do so.

It's a shame that Docker Engine API is limited and jib:dockerBuild has to load the entire image.

[chanseokoh]

Sorry for the late update. Re: putting arbitrary files into to the image, please see this doc.

If you need, you can leverage other plugins to dynamically download files online and put them into one of the <extraDirectories> at compile-time. Here'a an example to download a Cloud Java Debugger agent.

Regarding why you can't do Dockerfile's RUN-like things in Jib, I'll quote the following comments (from #1806 (comment) and #2249 (comment)).

the way Jib builds an image is fundamentally different from how the Docker CLI builds an image using Dockerfile (reproducible vs. non-reproducible, declarative vs. imperative, Docker and Dockerfile-less build vs. requiring Docker daemon and client, requiring root-privilege vs. not). Jib is in a very different realm, so unfortunately, it is very difficult to support ONBUILD unless we radically change our opinionated philosophy in how an image should be built. Basically, we don't "run" Dockerfile directives, particularly the ones like RUN that executes something. Jib doesn't provide/include a Docker runtime (that is one of the points of Jib).

And as for running arbitrary commands, unfortunately this is largely incompatible with the mode Jib operates in, because the way Jib builds an image is fundamentally different from how Docker does: #1806 (comment) We build images in a declarative and reproducible way without actually requiring to have a runtime component to be able to run an image at image build-time; running an image basically destroys reproducibility. So unfortunately it is very difficult for Jib to support "running" arbitrary commands at image building-time.