Skip to content

Latest commit

 

History

History
769 lines (695 loc) · 27.6 KB

docker-kubernetes.adoc

File metadata and controls

769 lines (695 loc) · 27.6 KB

Deploy Application on Kubernetes Cluster

Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
— github.com/GoogleCloudPlatform/kubernetes/

Kubernetes, or “k8s” in short, allows the user to provide declarative primitives for the desired state, for example “need 5 WildFly servers and 1 MySQL server running”. Kubernetes self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers then ensure that this state is met. The user just define the state and Kubernetes ensures that the state is met at all times on the cluster.

How is it related to Docker?

Docker provides the lifecycle management of containers. A Docker image defines a build time representation of the runtime containers. There are commands to start, stop, restart, link, and perform other lifecycle methods on these containers. Kubernetes uses Docker to package, instantiate, and run containerized applications.

How does Kubernetes simplify containerized application deployment?

A typical application would have a cluster of containers across multiple hosts. For example, your web tier (for example Apache) might run as a few instances, and likely on a set of containers. Similarly, your application tier (for example, WildFly) would run on a different set of containers. The web tier would need to delegate the request to application tier. The web, application, and database tier would generally run on a separate set of containers. These containers would need to talk to each other. Using any of the solutions mentioned above would require scripting to start the containers, and monitoring/bouncing if something goes down. Kubernetes does all of that for the user after the application state has been defined.

Key Components

At a very high level, there are three key components:

  1. Pods are the smallest deployable units that can be created, scheduled, and managed. Its a logical collection of containers that belong to an application.

  2. Master is the central control point that provides a unified view of the cluster. There is a single master node that control multiple worker nodes.

  3. Node (née minion) is a worker node that run tasks as delegated by the master. Nodes can run one or more pods. It provides an application-specific “virtual host” in a containerized environment.

A picture is always worth a thousand words and so this is a high-level logical block diagram for Kubernetes:

kubernetes key components
Figure 1. Kubernetes Key Components

After the 50,000 feet view, lets fly a little lower at 30,000 feet and take a look at how Kubernetes make all of this happen. There are a few key components at Master and Node that make this happen.

  1. Replication Controller is a resource at Master that ensures that requested number of pods are running on nodes at all times.

  2. Service is an object on master that provides load balancing across a replicated group of pods. Label is an arbitrary key/value pair in a distributed watchable storage that the Replication Controller uses for service discovery.

  3. Kubelet Each node runs services to run containers and be managed from the master. In addition to Docker, Kubelet is another key service installed there. It reads container manifests as YAML files that describes a pod. Kubelet ensures that the containers defined in the pods are started and continue running.

  4. Master serves RESTful Kubernetes API that validate and configure Pod, Service, and Replication Controller.

Start Kubernetes Cluster

Kubernetes cluster can be easily started using Vagrant. Start a Kubernetes cluster as explained in [Kubernetes_Setup].

Verify the Cluster

  1. Verify the Kubernetes cluster as:

    vagrant status
    Current machine states:
    
    master                    running (virtualbox)
    minion-1                  running (virtualbox)
    
    This environment represents multiple VMs. The VMs are all listed
    above with their current state. For more information about a specific
    VM, run `vagrant status NAME`.

    By default, the Vagrant setup will create a single Master and one node. Each VM will take 1 GB, so make sure you have at least 2GB to 4GB of free memory (plus appropriate free disk space).

    Note
    By default, only one node is created. This can be manipulated by setting an environment variable NUM_MINIONS variable to an integer before invoking kube-up.sh script.
    kubernetes cluster vagrant
    Figure 2. Kubernetes Cluster using Vagrant

    By default, each VM in the cluster is running Fedora, Kubelet is installed into ``systemd'', and all other Kubernetes services are running as containers on Master.

  2. Access https://10.245.1.2 (or whatever IP address is assigned to your kubernetes cluster start up log). This may present the warning as shown below:

    kubernetes master default output certificate

    Click on ‘Advanced’, on ‘Proceed to 10.245.1.2’, enter the username as ‘vagrant’ and password as ‘vagrant’ to see the output as:

    kubernetes master default output
    Figure 3. Kubernetes Output from Master

    Check the list of nodes as:

    ./cluster/kubectl.sh get nodes
    NAME         LABELS                              STATUS    AGE
    10.245.1.3   kubernetes.io/hostname=10.245.1.3   Ready     1m
  3. Check the list of pods:

    ./cluster/kubectl.sh get po
    NAME      READY     STATUS    RESTARTS   AGE
  4. Check the list of services running:

    ./cluster/kubectl.sh get svc
    NAME         CLUSTER_IP   EXTERNAL_IP   PORT(S)   SELECTOR   AGE
    kubernetes   10.247.0.1   <none>        443/TCP   <none>     3m
  5. Check the list of replication controllers:

    ./cluster/kubectl.sh get rc
    CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR   REPLICAS

Deploy Java EE Application (multiple configuration files)

Pods, and the IP addresses assigned to them, are ephemeral. If a pod dies then Kubernetes will recreate that pod because of its self-healing features, but it might recreate it on a different host. Even if it is on the same host, a different IP address could be assigned to it. And so any application cannot rely upon the IP address of the pod.

Kubernetes services is an abstraction which defines a logical set of pods. A service is typically back-ended by one or more physical pods (associated using labels), and it has a permanent IP address that can be used by other pods/applications. For example, WildFly pod can not directly connect to a Couchbase pod but can connect to Couchbase service. In essence, Kubernetes service offers clients an IP and port pair which, when accessed, redirects to the appropriate backends.

kubernetes service
Figure 4. Kubernetes Service
Note
In this case, all the pods are running on a single node. This is because, that is the default number for a Kubernetes cluster. The pod can be on another node if more number of nodes are configured to start in the cluster.

Any Service that a Pod wants to access must be created before the Pod itself, or else the environment variables will not be populated.

The order of Service and the targeted Pods does not matter. However Service needs to be started before any other Pods consuming the Service are started.

Start Couchbase Pod

  1. Start Couchbase Pod:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app-couchbase-pod.yaml
    pod "couchbase-pod" created

    It uses the following configuration file:

    apiVersion: v1
    kind: Pod
    metadata:
      name: couchbase-pod
      labels:
        name: couchbase-pod
        context: docker-k8s-lab
    spec:
      containers:
        -
          name: couchbase
          image: arungupta/couchbase
          ports:
            -
              containerPort: 8091
              containerPort: 8092
              containerPort: 8093
              containerPort: 11210

    arungupta/couchbase image starts Couchbase server, configures it using Couchbase REST API, and loads a sample data bucket.

  2. Get status of the Pod:

    ./cluster/kubectl.sh get -w po
    NAME            READY     STATUS    RESTARTS   AGE
    couchbase-pod   0/1       Pending   0          16s
    NAME            READY     STATUS    RESTARTS   AGE
    couchbase-pod   0/1       Running   0          40s
    couchbase-pod   1/1       Running   0         40s

    -w watches for changes to the requested object. Wait for the Couchbase pod to be in Running status.

    Hit Ctrl + C to terminate the watch.

Start Couchbase service

  1. Start Couchbase Service:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app-couchbase-service.yaml
    service "couchbase-service" created

    It uses the following configuration file:

    apiVersion: v1
    kind: Service
    metadata:
      name: couchbase-service
      labels:
        name: couchbase-pod
        context: docker-k8s-lab
    spec:
      ports:
        # the port that this service should serve on
        - name: admin
          port: 8091
        - name: queries
          port: 8092
        - name: rest
          port: 8093
        - name: nodes
          port: 11210
      # label keys and values that must match in order to receive traffic for this service
      selector:
        name: couchbase-pod
        context: docker-k8s-lab

    Once again, the label “context: docker-k8s-lab” is used. This simplifies querying the created pods later on.

  2. Get status of the Service:

    ./cluster/kubectl.sh get svc
    NAME                CLUSTER_IP      EXTERNAL_IP   PORT(S)                                SELECTOR                                    AGE
    couchbase-service   10.247.202.53   <none>        8091/TCP,8092/TCP,8093/TCP,11210/TCP   context=docker-k8s-lab,name=couchbase-pod   1m
    kubernetes          10.247.0.1      <none>        443/TCP                                <none>                                      32m

    If multiple services are running, then it can be narrowed by specifying the labels:

    ./cluster/kubectl.sh get svc -l context=docker-k8s-lab,name=couchbase-pod
    NAME                CLUSTER_IP      EXTERNAL_IP   PORT(S)                                SELECTOR                                    AGE
    couchbase-service   10.247.202.53   <none>        8091/TCP,8092/TCP,8093/TCP,11210/TCP   context=docker-k8s-lab,name=couchbase-pod   1m

    This is also the selector label used by Service to target Pods.

    When a Service is run on a node, the kubelet adds a set of environment variables for each active Service. It supports both Docker links compatible variables and simpler {SVCNAME}_SERVICE_HOST and {SVCNAME}_SERVICE_PORT variables, where the Service name is upper-cased and dashes are converted to underscores.

    Our service name is “couchbase-service” and so COUCHBASE_SERVICE_SERVICE_HOST and COUCHBASE_SERVICE_SERVICE_PORT variables are available to other pods.

Kubernetes also allows services to be resolved using DNS configuration. Send a Pull Request for adding this functionality to the lab as explained in #62.

Start WildFly Replication Controller

  1. Start WildFly replication controller:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app-wildfly-rc.yaml
    replicationcontroller "wildfly-rc" created

    It uses the following configuration file:

    apiVersion: v1
    kind: ReplicationController
    metadata:
      name: wildfly-rc
      labels:
        name: wildfly
        context: docker-k8s-lab
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            name: wildfly
        spec:
          containers:
          - name: wildfly-rc-pod
            image: arungupta/wildfly-couchbase-javaee7
            env:
              - name: COUCHBASE_URI
                value: $COUCHBASE_SERVICE_SERVICE_HOST
            ports:
            - containerPort: 8080
  2. Check status of the Replication Controller:

    ./cluster/kubectl.sh get rc
    CONTROLLER   CONTAINER(S)     IMAGE(S)                              SELECTOR       REPLICAS   AGE
    wildfly-rc   wildfly-rc-pod   arungupta/wildfly-mysql-javaee7:k8s   name=wildfly   1          1m
  3. Check status of the Pod inside Replication Controller:

    ./cluster/kubectl.sh get -w po
    NAME               READY     STATUS    RESTARTS   AGE
    mysql-pod          1/1       Running   0          4m
    wildfly-rc-ca1ug   0/1       Pending   0          1m
    NAME               READY     STATUS    RESTARTS   AGE
    wildfly-rc-ca1ug   0/1       Running   0          3m
    wildfly-rc-ca1ug   1/1       Running   0         3m
  4. Use the Pod’s name to get IP address:

    ./cluster/kubectl.sh get -o template po wildfly-rc-ca1ug --template={{.status.podIP}}
    10.246.96.7

Access the application (using node)

  1. Log in to node:

    vagrant ssh minion-1
  2. Access the application using curl http://10.246.96.7:8080/employees/resources/employees and replace IP address with the one obtained earlier:

    > vagrant ssh minion-1
    Last login: Sat Nov 21 01:02:52 2015 from 10.0.2.2
    [vagrant@kubernetes-minion-1 ~]$ curl http://10.246.96.7:8080/employees/resources/employees
    <?xml version="1.0" encoding="UTF-8" standalone="yes"?><collection><employee><id>1</id><name>Penny</name></employee><employee><id>2</id><name>Sheldon</name></employee><employee><id>3</id><name>Amy</name></employee><employee><id>4</id><name>Leonard</name></employee><employee><id>5</id><name>Bernadette</name></employee><employee><id>6</id><name>Raj</name></employee><employee><id>7</id><name>Howard</name></employee><employee><id>8</id><name>Priya</name></employee></collection>[vagrant@kubernetes-minion-1 ~]
  3. Log out of the minion

    [vagrant@kubernetes-minion-1 ~]$ exit
    logout
    Connection to 127.0.0.1 closed.

Access the application (using proxy)

Send a PR for arun-gupta#80

Deploy Java EE Application (one configuration file)

Kubernetes allow multiple resources to be specified in a single configuration file. This allows to create a “Kubernetes Application” that can consists of multiple resources easily.

Deploy Java EE Application (multiple configuration files) showed how to deploy the Java EE application using multiple configuration files. This application can be delpoyed using a single configuration file as well.

  1. First delete the the existing pods and replication controller using the context label:

    ./cluster/kubectl.sh delete svc,po,replicationController -l context="docker-k8s-lab"
    service "mysql-service" deleted
    pod "mysql-pod" deleted
    replicationcontroller "wildfly-rc" deleted
  2. Start the application using the configuration file:

    apiVersion: v1
    kind: Pod
    metadata:
      name: mysql-pod
      labels:
        name: mysql-pod
        context: docker-k8s-lab
    spec:
      containers:
        -
          name: mysql
          image: mysql:latest
          env:
            -
              name: "MYSQL_USER"
              value: "mysql"
            -
              name: "MYSQL_PASSWORD"
              value: "mysql"
            -
              name: "MYSQL_DATABASE"
              value: "sample"
            -
              name: "MYSQL_ROOT_PASSWORD"
              value: "supersecret"
          ports:
            -
              containerPort: 3306
    ----
    apiVersion: v1
    kind: Service
    metadata:
      name: mysql-service
      labels:
        name: mysql-pod
        context: docker-k8s-lab
    spec:
      ports:
        # the port that this service should serve on
        - port: 3306
      # label keys and values that must match in order to receive traffic for this service
      selector:
        name: mysql-pod
        context: docker-k8s-lab
    ----
    apiVersion: v1
    kind: ReplicationController
    metadata:
      name: wildfly-rc
      labels:
        name: wildfly
        context: docker-k8s-lab
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            name: wildfly
        spec:
          containers:
          - name: wildfly-rc-pod
            image: arungupta/wildfly-mysql-javaee7:k8s
            ports:
            - containerPort: 8080

    Notice that each section, one each for MySQL Pod, MySQL Service, and WildFly Replication Controller, is separated by ----.

  3. Start the application:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app.yaml
    pod "mysql-pod" created
    service "mysql-service" created
    replicationcontroller "wildfly-rc" created
  4. Access the application using Access the application (using node) or Access the application (using proxy).

Rescheduling Pods

Replication Controller ensures that specified number of pod “replicas” are running at any one time. If there are too many, the replication controller kills some pods. If there are too few, it starts more.

WildFly Replication Controller is already running with one Pod. Lets delete this Pod and see how a new Pod is automatically rescheduled.

  1. Find the Pod’s name:

    ./cluster/kubectl.sh get po
    NAME               READY     STATUS    RESTARTS   AGE
    mysql-pod          1/1       Running   0          21s
    wildfly-rc-l2cto   1/1       Running   0          21s
  2. Delete the Pod:

    ./cluster/kubectl.sh delete po wildfly-rc-l2cto
    pod "wildfly-rc-l2cto" deleted

    Status of the Pods can be seen:

    ./cluster/kubectl.sh get -w po
    NAME               READY     STATUS        RESTARTS   AGE
    mysql-pod          1/1       Running       0          1m
    wildfly-rc-2o8vd   1/1       Running       0          13s
    wildfly-rc-l2cto   1/1       Terminating   0          1m

    Notice how Pod with name “wildfly-rc-l2cto” was deleted and a new Pod with the name “wildfly-rc-2o8vd” was created. The status “Terminating” does not update correctly and is filed as kubernetes/kubernetes#17612.

Scaling Pods

Replication Controller allows dynamic scaling up and down of Pods.

  1. Scale up the number of Pods:

    ./cluster/kubectl.sh scale --replicas=2 rc wildfly-rc
    replicationcontroller "wildfly-rc" scaled
  2. Status of the Pods can be seen in another shell:

    ./cluster/kubectl.sh get -w po
    TBD

    Pods are not scaled correctly as explained at kubernetes/kubernetes#17613.

    TBD: Notice a new Pod with the name “wildfly-rc-bymu7” is created.

  3. Scale down the number of Pods:

    ./cluster/kubectl.sh scale --replicas=1 rc wildfly-rc
    scaled
  4. Status of the Pods using -w is not shown correctly #11338. But status of the Pods can be seen correctly as:

    ./cluster/kubectl.sh get po
    NAME               READY     STATUS    RESTARTS   AGE
    wildfly-rc-bgtkg   1/1       Running   0          9m

    Notice only one Pod is running now.

Couchbase Cluster and Persistent Volumes

A Kubernetes Volume outlives any containers that run within the Pod, and data is preserved across Container restarts. However the volume will cease to exist when a Pod ceases to exist. This is solved by Persistent Volumes that provide persistent, cluster-scoped storage for applications that require long lived data.

Creating and using a persistent volume is a three step process:

  1. Administrator provision a networked storage in the cluster. This is called as PersistentVolume.

  2. User requests storage for pods by using claims. Claims can specify levels of resources (CPU and memory), specific sizes and access modes (e.g. can be mounted once read/write or many times write only). This is called as PersistentVolumeClaim.

  3. Claims are mounted as volumes and used in pods for storage.

Create Kubernetes resources

  1. Create persistent volume:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-pv.yaml
    persistentvolume "pv001" created
  2. Create a claim for volume:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-pvc.yaml
    persistentvolumeclaim "pvc001" created
  3. Start the Couchbase Replication Controller (claim is used in Pod’s definition):

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-rc.yaml
    replicationcontroller "couchbase-rc" created

    Check status of Replication Controller:

    ./cluster/kubectl.sh get -w po
    NAME                 READY     STATUS    RESTARTS   AGE
    couchbase-rc-s5o22   0/1       Pending   0          9s
    NAME                 READY     STATUS    RESTARTS   AGE
    couchbase-rc-s5o22   0/1       Running   0          47s
    couchbase-rc-s5o22   1/1       Running   0         47s
  4. Start Couchbase Service:

    ./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-service.yaml
    service "couchbase-service" created

Initialize Couchbase Cluster

  1. Set data and index memory quota for Couchbase:

    ./cluster/kubectl.sh exec couchbase-rc-s5o22 -- curl -v -X POST http://localhost:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300
  2. Configure Data and Query service on Couchbase server:

    ./cluster/kubectl.sh exec couchbase-rc-s5o22 -- /opt/couchbase/bin/couchbase-cli cluster-init -u Administrator -p password -c localhost:8091 --cluster-ramsize=256 --cluster-index-ramsize=256 --services=index,data,query
    SUCCESS: init/edit localhost
  3. Install travel-sample bucket:

    ./cluster/kubectl.sh exec couchbase-rc-s5o22 -- curl -v -u Administrator:password -X POST http://localhost:8091/sampleBuckets/install -d '["travel-sample"]'
    *   Trying ::1...
    * connect to ::1 port 8091 failed: Connection refused
    *   Trying 127.0.0.1...
    * Connected to localhost (127.0.0.1) port 8091 (#0)
    * Server auth using Basic with user 'Administrator'
    > POST /sampleBuckets/install HTTP/1.1
    > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==
    > User-Agent: curl/7.40.0-DEV
    > Host: localhost:8091
    > Accept: */*
    > Content-Length: 17
    > Content-Type: application/x-www-form-urlencoded
    >
    * upload completely sent off: 17 out of 17 bytes
    < HTTP/1.1 202 Accepted
    < Server: Couchbase Server
    < Pragma: no-cache
    < Date: Mon, 23 Nov 2015 04:31:22 GMT
    < Content-Type: application/json
    < Content-Length: 2
    < Cache-Control: no-cache
    <
    * Connection #0 to host localhost left intact
  4. Query using cbq gives error:

    ./cluster/kubectl.sh exec -it couchbase-rc-s5o22 -- /opt/couchbase/bin/cbq

    shows the message:

    Couchbase query shell connected to http://localhost:8093/ . Type Ctrl-D to exit.

Add a new Couchbase server to the cluster

  1. Scale using RC

Application Logs

  1. Get a list of the Pods:

    ./cluster/kubectl.sh get po
    NAME               READY     STATUS    RESTARTS   AGE
    mysql-pod          1/1       Running   0          18h
    wildfly-rc-w2kk5   1/1       Running   0          16h
  2. Get logs for the WildFly Pod:

    ./cluster/kubectl.sh logs wildfly-rc-w2kk5
    => Starting WildFly server
    => Waiting for the server to boot
    =========================================================================
    
      JBoss Bootstrap Environment
    
      JBOSS_HOME: /opt/jboss/wildfly
    
      . . .

Logs can be obtained for any Kubernetes resources using this way. Alternatively, the logs can also be seen by logging into the node:

  1. Log in to the node VM:

    > vagrant ssh minion-1
    Last login: Fri Jun  5 23:01:36 2015 from 10.0.2.2
    [vagrant@kubernetes-minion-1 ~]$
  2. Log in as root:

    [vagrant@kubernetes-minion-1 ~]$ su -
    Password:
    [root@kubernetes-minion-1 ~]#

    Default root password for VM images created by Vagrant is ‘vagrant’.

  3. See the list of Docker containers running on this VM:

    docker ps
  4. View WildFly log as:

    docker logs $(docker ps | grep arungupta/wildfly | awk '{print $1}')
  5. View MySQL log as:

    docker logs <CID>

Delete Kubernetes Resources

Individual resources (service, replication controller, or pod) can be deleted by using delete command instead of create command. Alternatively, all services and replication controllers can be deleted using a label as:

kubectl delete -l se,po context=docker-k8s-lab

Stop Kubernetes Cluster

> ./cluster/kube-down.sh
Bringing down cluster using provider: vagrant
==> minion-1: Forcing shutdown of VM...
==> minion-1: Destroying VM and associated drives...
==> master: Forcing shutdown of VM...
==> master: Destroying VM and associated drives...
Done

Debug Kubernetes Master

  1. Log in to the master as:

    vagrant ssh master
    Last login: Wed Jul 15 20:36:32 2015 from 10.0.2.2
    [vagrant@kubernetes-master ~]$
  2. Log in as root:

    [vagrant@kubernetes-master ~]$ su -
    Password:
    [root@kubernetes-master ~]#

    Default root password for VM images created by Vagrant is ‘vagrant’.

  3. Check the containers running on master:

    CONTAINER ID        IMAGE                                                                               COMMAND                CREATED             STATUS              PORTS               NAMES
    dc59a764953c        gcr.io/google_containers/etcd:2.0.12                                                "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_etcd-container.fa2ab1d9_etcd-server-kubernetes-master_default_7b64ecafde589b94a342982699601a19_2b69c4d5
    b722e22d3ddb        gcr.io/google_containers/kube-scheduler:d1107ff3b8fcdcbf5a9d78d9d6dbafb1            "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_kube-scheduler.7501c229_kube-scheduler-kubernetes-master_default_98b354f725c1589ea5a12119795546ae_b81b9740
    38a73e342866        gcr.io/google_containers/kube-controller-manager:fafaf8100ccc963e643b55e35386d713   "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_kube-controller-manager.db050993_kube-controller-manager-kubernetes-master_default_f5c25224fbfb2de87e1e5c35e6b3a293_dcd4cb5d
    01001de6409e        gcr.io/google_containers/kube-apiserver:cff9e185796caa8b281e7d961aea828b            "/bin/sh -c '/usr/lo   20 hours ago        Up 20 hours                             k8s_kube-apiserver.7e06f4e1_kube-apiserver-kubernetes-master_default_829f8c23fd5fc7951253cac7618447fc_b39c0a5d
    0f8ccb144ece        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_kube-scheduler-kubernetes-master_default_98b354f725c1589ea5a12119795546ae_eb1efcac
    0b8f527456c0        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_kube-apiserver-kubernetes-master_default_829f8c23fd5fc7951253cac7618447fc_5dd4dee7
    39d9c41ab1a2        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_kube-controller-manager-kubernetes-master_default_f5c25224fbfb2de87e1e5c35e6b3a293_522972ae
    d970ddff7046        gcr.io/google_containers/pause:0.8.0                                                "/pause"               20 hours ago        Up 20 hours                             k8s_POD.e4cc795_etcd-server-kubernetes-master_default_7b64ecafde589b94a342982699601a19_fa75b27f