Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications.
Kubernetes, or “k8s” in short, allows the user to provide declarative primitives for the desired state, for example “need 5 WildFly servers and 1 MySQL server running”. Kubernetes self-healing mechanisms, such as auto-restarting, re-scheduling, and replicating containers then ensure that this state is met. The user just define the state and Kubernetes ensures that the state is met at all times on the cluster.
How is it related to Docker?
Docker provides the lifecycle management of containers. A Docker image defines a build time representation of the runtime containers. There are commands to start, stop, restart, link, and perform other lifecycle methods on these containers. Kubernetes uses Docker to package, instantiate, and run containerized applications.
How does Kubernetes simplify containerized application deployment?
A typical application would have a cluster of containers across multiple hosts. For example, your web tier (for example Apache) might run as a few instances, and likely on a set of containers. Similarly, your application tier (for example, WildFly) would run on a different set of containers. The web tier would need to delegate the request to application tier. The web, application, and database tier would generally run on a separate set of containers. These containers would need to talk to each other. Using any of the solutions mentioned above would require scripting to start the containers, and monitoring/bouncing if something goes down. Kubernetes does all of that for the user after the application state has been defined.
At a very high level, there are three key components:
-
Pods are the smallest deployable units that can be created, scheduled, and managed. Its a logical collection of containers that belong to an application.
-
Master is the central control point that provides a unified view of the cluster. There is a single master node that control multiple worker nodes.
-
Node (née minion) is a worker node that run tasks as delegated by the master. Nodes can run one or more pods. It provides an application-specific “virtual host” in a containerized environment.
A picture is always worth a thousand words and so this is a high-level logical block diagram for Kubernetes:
After the 50,000 feet view, lets fly a little lower at 30,000 feet and take a look at how Kubernetes make all of this happen. There are a few key components at Master and Node that make this happen.
-
Replication Controller is a resource at Master that ensures that requested number of pods are running on nodes at all times.
-
Service is an object on master that provides load balancing across a replicated group of pods. Label is an arbitrary key/value pair in a distributed watchable storage that the Replication Controller uses for service discovery.
-
Kubelet Each node runs services to run containers and be managed from the master. In addition to Docker, Kubelet is another key service installed there. It reads container manifests as YAML files that describes a pod. Kubelet ensures that the containers defined in the pods are started and continue running.
-
Master serves RESTful Kubernetes API that validate and configure Pod, Service, and Replication Controller.
Kubernetes cluster can be easily started using Vagrant. Start a Kubernetes cluster as explained in [Kubernetes_Setup].
-
Verify the Kubernetes cluster as:
vagrant status Current machine states: master running (virtualbox) minion-1 running (virtualbox) This environment represents multiple VMs. The VMs are all listed above with their current state. For more information about a specific VM, run `vagrant status NAME`.
By default, the Vagrant setup will create a single Master and one node. Each VM will take 1 GB, so make sure you have at least 2GB to 4GB of free memory (plus appropriate free disk space).
NoteBy default, only one node is created. This can be manipulated by setting an environment variable NUM_MINIONS variable to an integer before invoking kube-up.sh
script.By default, each VM in the cluster is running Fedora, Kubelet is installed into ``systemd'', and all other Kubernetes services are running as containers on Master.
-
Access https://10.245.1.2 (or whatever IP address is assigned to your kubernetes cluster start up log). This may present the warning as shown below:
Click on ‘Advanced’, on ‘Proceed to 10.245.1.2’, enter the username as ‘vagrant’ and password as ‘vagrant’ to see the output as:
Check the list of nodes as:
./cluster/kubectl.sh get nodes NAME LABELS STATUS AGE 10.245.1.3 kubernetes.io/hostname=10.245.1.3 Ready 1m
-
Check the list of pods:
./cluster/kubectl.sh get po NAME READY STATUS RESTARTS AGE
-
Check the list of services running:
./cluster/kubectl.sh get svc NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE kubernetes 10.247.0.1 <none> 443/TCP <none> 3m
-
Check the list of replication controllers:
./cluster/kubectl.sh get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
Pods, and the IP addresses assigned to them, are ephemeral. If a pod dies then Kubernetes will recreate that pod because of its self-healing features, but it might recreate it on a different host. Even if it is on the same host, a different IP address could be assigned to it. And so any application cannot rely upon the IP address of the pod.
Kubernetes services is an abstraction which defines a logical set of pods. A service is typically back-ended by one or more physical pods (associated using labels), and it has a permanent IP address that can be used by other pods/applications. For example, WildFly pod can not directly connect to a Couchbase pod but can connect to Couchbase service. In essence, Kubernetes service offers clients an IP and port pair which, when accessed, redirects to the appropriate backends.
Note
|
In this case, all the pods are running on a single node. This is because, that is the default number for a Kubernetes cluster. The pod can be on another node if more number of nodes are configured to start in the cluster. |
Any Service that a Pod wants to access must be created before the Pod itself, or else the environment variables will not be populated.
The order of Service and the targeted Pods does not matter. However Service needs to be started before any other Pods consuming the Service are started.
-
Start Couchbase Pod:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app-couchbase-pod.yaml pod "couchbase-pod" created
It uses the following configuration file:
apiVersion: v1 kind: Pod metadata: name: couchbase-pod labels: name: couchbase-pod context: docker-k8s-lab spec: containers: - name: couchbase image: arungupta/couchbase ports: - containerPort: 8091 containerPort: 8092 containerPort: 8093 containerPort: 11210
arungupta/couchbase
image starts Couchbase server, configures it using Couchbase REST API, and loads a sample data bucket. -
Get status of the Pod:
./cluster/kubectl.sh get -w po NAME READY STATUS RESTARTS AGE couchbase-pod 0/1 Pending 0 16s NAME READY STATUS RESTARTS AGE couchbase-pod 0/1 Running 0 40s couchbase-pod 1/1 Running 0 40s
-w
watches for changes to the requested object. Wait for the Couchbase pod to be in Running status.Hit
Ctrl
+C
to terminate the watch.
-
Start Couchbase Service:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app-couchbase-service.yaml service "couchbase-service" created
It uses the following configuration file:
apiVersion: v1 kind: Service metadata: name: couchbase-service labels: name: couchbase-pod context: docker-k8s-lab spec: ports: # the port that this service should serve on - name: admin port: 8091 - name: queries port: 8092 - name: rest port: 8093 - name: nodes port: 11210 # label keys and values that must match in order to receive traffic for this service selector: name: couchbase-pod context: docker-k8s-lab
Once again, the label “context: docker-k8s-lab” is used. This simplifies querying the created pods later on.
-
Get status of the Service:
./cluster/kubectl.sh get svc NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE couchbase-service 10.247.202.53 <none> 8091/TCP,8092/TCP,8093/TCP,11210/TCP context=docker-k8s-lab,name=couchbase-pod 1m kubernetes 10.247.0.1 <none> 443/TCP <none> 32m
If multiple services are running, then it can be narrowed by specifying the labels:
./cluster/kubectl.sh get svc -l context=docker-k8s-lab,name=couchbase-pod NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE couchbase-service 10.247.202.53 <none> 8091/TCP,8092/TCP,8093/TCP,11210/TCP context=docker-k8s-lab,name=couchbase-pod 1m
This is also the selector label used by Service to target Pods.
When a Service is run on a node, the kubelet adds a set of environment variables for each active Service. It supports both Docker links compatible variables and simpler
{SVCNAME}_SERVICE_HOST
and{SVCNAME}_SERVICE_PORT
variables, where the Service name is upper-cased and dashes are converted to underscores.Our service name is “couchbase-service” and so
COUCHBASE_SERVICE_SERVICE_HOST
andCOUCHBASE_SERVICE_SERVICE_PORT
variables are available to other pods.
Kubernetes also allows services to be resolved using DNS configuration. Send a Pull Request for adding this functionality to the lab as explained in #62.
-
Start WildFly replication controller:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app-wildfly-rc.yaml replicationcontroller "wildfly-rc" created
It uses the following configuration file:
apiVersion: v1 kind: ReplicationController metadata: name: wildfly-rc labels: name: wildfly context: docker-k8s-lab spec: replicas: 1 template: metadata: labels: name: wildfly spec: containers: - name: wildfly-rc-pod image: arungupta/wildfly-couchbase-javaee7 env: - name: COUCHBASE_URI value: $COUCHBASE_SERVICE_SERVICE_HOST ports: - containerPort: 8080
-
Check status of the Replication Controller:
./cluster/kubectl.sh get rc CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS AGE wildfly-rc wildfly-rc-pod arungupta/wildfly-mysql-javaee7:k8s name=wildfly 1 1m
-
Check status of the Pod inside Replication Controller:
./cluster/kubectl.sh get -w po NAME READY STATUS RESTARTS AGE mysql-pod 1/1 Running 0 4m wildfly-rc-ca1ug 0/1 Pending 0 1m NAME READY STATUS RESTARTS AGE wildfly-rc-ca1ug 0/1 Running 0 3m wildfly-rc-ca1ug 1/1 Running 0 3m
-
Use the Pod’s name to get IP address:
./cluster/kubectl.sh get -o template po wildfly-rc-ca1ug --template={{.status.podIP}} 10.246.96.7
-
Log in to node:
vagrant ssh minion-1
-
Access the application using
curl http://10.246.96.7:8080/employees/resources/employees
and replace IP address with the one obtained earlier:> vagrant ssh minion-1 Last login: Sat Nov 21 01:02:52 2015 from 10.0.2.2 [vagrant@kubernetes-minion-1 ~]$ curl http://10.246.96.7:8080/employees/resources/employees <?xml version="1.0" encoding="UTF-8" standalone="yes"?><collection><employee><id>1</id><name>Penny</name></employee><employee><id>2</id><name>Sheldon</name></employee><employee><id>3</id><name>Amy</name></employee><employee><id>4</id><name>Leonard</name></employee><employee><id>5</id><name>Bernadette</name></employee><employee><id>6</id><name>Raj</name></employee><employee><id>7</id><name>Howard</name></employee><employee><id>8</id><name>Priya</name></employee></collection>[vagrant@kubernetes-minion-1 ~]
-
Log out of the minion
[vagrant@kubernetes-minion-1 ~]$ exit logout Connection to 127.0.0.1 closed.
Send a PR for arun-gupta#80
Kubernetes allow multiple resources to be specified in a single configuration file. This allows to create a “Kubernetes Application” that can consists of multiple resources easily.
Deploy Java EE Application (multiple configuration files) showed how to deploy the Java EE application using multiple configuration files. This application can be delpoyed using a single configuration file as well.
-
First delete the the existing pods and replication controller using the context label:
./cluster/kubectl.sh delete svc,po,replicationController -l context="docker-k8s-lab" service "mysql-service" deleted pod "mysql-pod" deleted replicationcontroller "wildfly-rc" deleted
-
Start the application using the configuration file:
apiVersion: v1 kind: Pod metadata: name: mysql-pod labels: name: mysql-pod context: docker-k8s-lab spec: containers: - name: mysql image: mysql:latest env: - name: "MYSQL_USER" value: "mysql" - name: "MYSQL_PASSWORD" value: "mysql" - name: "MYSQL_DATABASE" value: "sample" - name: "MYSQL_ROOT_PASSWORD" value: "supersecret" ports: - containerPort: 3306 ---- apiVersion: v1 kind: Service metadata: name: mysql-service labels: name: mysql-pod context: docker-k8s-lab spec: ports: # the port that this service should serve on - port: 3306 # label keys and values that must match in order to receive traffic for this service selector: name: mysql-pod context: docker-k8s-lab ---- apiVersion: v1 kind: ReplicationController metadata: name: wildfly-rc labels: name: wildfly context: docker-k8s-lab spec: replicas: 1 template: metadata: labels: name: wildfly spec: containers: - name: wildfly-rc-pod image: arungupta/wildfly-mysql-javaee7:k8s ports: - containerPort: 8080
Notice that each section, one each for MySQL Pod, MySQL Service, and WildFly Replication Controller, is separated by
----
. -
Start the application:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/app.yaml pod "mysql-pod" created service "mysql-service" created replicationcontroller "wildfly-rc" created
-
Access the application using Access the application (using node) or Access the application (using proxy).
Replication Controller ensures that specified number of pod “replicas” are running at any one time. If there are too many, the replication controller kills some pods. If there are too few, it starts more.
WildFly Replication Controller is already running with one Pod. Lets delete this Pod and see how a new Pod is automatically rescheduled.
-
Find the Pod’s name:
./cluster/kubectl.sh get po NAME READY STATUS RESTARTS AGE mysql-pod 1/1 Running 0 21s wildfly-rc-l2cto 1/1 Running 0 21s
-
Delete the Pod:
./cluster/kubectl.sh delete po wildfly-rc-l2cto pod "wildfly-rc-l2cto" deleted
Status of the Pods can be seen:
./cluster/kubectl.sh get -w po NAME READY STATUS RESTARTS AGE mysql-pod 1/1 Running 0 1m wildfly-rc-2o8vd 1/1 Running 0 13s wildfly-rc-l2cto 1/1 Terminating 0 1m
Notice how Pod with name “wildfly-rc-l2cto” was deleted and a new Pod with the name “wildfly-rc-2o8vd” was created. The status “Terminating” does not update correctly and is filed as kubernetes/kubernetes#17612.
Replication Controller allows dynamic scaling up and down of Pods.
-
Scale up the number of Pods:
./cluster/kubectl.sh scale --replicas=2 rc wildfly-rc replicationcontroller "wildfly-rc" scaled
-
Status of the Pods can be seen in another shell:
./cluster/kubectl.sh get -w po TBD
Pods are not scaled correctly as explained at kubernetes/kubernetes#17613.
TBD: Notice a new Pod with the name “wildfly-rc-bymu7” is created.
-
Scale down the number of Pods:
./cluster/kubectl.sh scale --replicas=1 rc wildfly-rc scaled
-
Status of the Pods using
-w
is not shown correctly #11338. But status of the Pods can be seen correctly as:./cluster/kubectl.sh get po NAME READY STATUS RESTARTS AGE wildfly-rc-bgtkg 1/1 Running 0 9m
Notice only one Pod is running now.
A Kubernetes Volume outlives any containers that run within the Pod, and data is preserved across Container restarts. However the volume will cease to exist when a Pod ceases to exist. This is solved by Persistent Volumes that provide persistent, cluster-scoped storage for applications that require long lived data.
Creating and using a persistent volume is a three step process:
-
Administrator provision a networked storage in the cluster. This is called as
PersistentVolume
. -
User requests storage for pods by using claims. Claims can specify levels of resources (CPU and memory), specific sizes and access modes (e.g. can be mounted once read/write or many times write only). This is called as
PersistentVolumeClaim
. -
Claims are mounted as volumes and used in pods for storage.
-
Create persistent volume:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-pv.yaml persistentvolume "pv001" created
-
Create a claim for volume:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-pvc.yaml persistentvolumeclaim "pvc001" created
-
Start the Couchbase Replication Controller (claim is used in Pod’s definition):
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-rc.yaml replicationcontroller "couchbase-rc" created
Check status of Replication Controller:
./cluster/kubectl.sh get -w po NAME READY STATUS RESTARTS AGE couchbase-rc-s5o22 0/1 Pending 0 9s NAME READY STATUS RESTARTS AGE couchbase-rc-s5o22 0/1 Running 0 47s couchbase-rc-s5o22 1/1 Running 0 47s
-
Start Couchbase Service:
./cluster/kubectl.sh create -f ~/workspaces/docker-java/attendees/kubernetes/couchbase-service.yaml service "couchbase-service" created
-
Set data and index memory quota for Couchbase:
./cluster/kubectl.sh exec couchbase-rc-s5o22 -- curl -v -X POST http://localhost:8091/pools/default -d memoryQuota=300 -d indexMemoryQuota=300
-
Configure Data and Query service on Couchbase server:
./cluster/kubectl.sh exec couchbase-rc-s5o22 -- /opt/couchbase/bin/couchbase-cli cluster-init -u Administrator -p password -c localhost:8091 --cluster-ramsize=256 --cluster-index-ramsize=256 --services=index,data,query SUCCESS: init/edit localhost
-
Install travel-sample bucket:
./cluster/kubectl.sh exec couchbase-rc-s5o22 -- curl -v -u Administrator:password -X POST http://localhost:8091/sampleBuckets/install -d '["travel-sample"]' * Trying ::1... * connect to ::1 port 8091 failed: Connection refused * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 8091 (#0) * Server auth using Basic with user 'Administrator' > POST /sampleBuckets/install HTTP/1.1 > Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA== > User-Agent: curl/7.40.0-DEV > Host: localhost:8091 > Accept: */* > Content-Length: 17 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 17 out of 17 bytes < HTTP/1.1 202 Accepted < Server: Couchbase Server < Pragma: no-cache < Date: Mon, 23 Nov 2015 04:31:22 GMT < Content-Type: application/json < Content-Length: 2 < Cache-Control: no-cache < * Connection #0 to host localhost left intact
-
Query using
cbq
gives error:./cluster/kubectl.sh exec -it couchbase-rc-s5o22 -- /opt/couchbase/bin/cbq
shows the message:
Couchbase query shell connected to http://localhost:8093/ . Type Ctrl-D to exit.
-
Get a list of the Pods:
./cluster/kubectl.sh get po NAME READY STATUS RESTARTS AGE mysql-pod 1/1 Running 0 18h wildfly-rc-w2kk5 1/1 Running 0 16h
-
Get logs for the WildFly Pod:
./cluster/kubectl.sh logs wildfly-rc-w2kk5 => Starting WildFly server => Waiting for the server to boot ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /opt/jboss/wildfly . . .
Logs can be obtained for any Kubernetes resources using this way. Alternatively, the logs can also be seen by logging into the node:
-
Log in to the node VM:
> vagrant ssh minion-1 Last login: Fri Jun 5 23:01:36 2015 from 10.0.2.2 [vagrant@kubernetes-minion-1 ~]$
-
Log in as root:
[vagrant@kubernetes-minion-1 ~]$ su - Password: [root@kubernetes-minion-1 ~]#
Default root password for VM images created by Vagrant is ‘vagrant’.
-
See the list of Docker containers running on this VM:
docker ps
-
View WildFly log as:
docker logs $(docker ps | grep arungupta/wildfly | awk '{print $1}')
-
View MySQL log as:
docker logs <CID>
Individual resources (service, replication controller, or pod) can be deleted by using delete
command instead of create
command. Alternatively, all services and replication controllers can be deleted using a label as:
kubectl delete -l se,po context=docker-k8s-lab
> ./cluster/kube-down.sh
Bringing down cluster using provider: vagrant
==> minion-1: Forcing shutdown of VM...
==> minion-1: Destroying VM and associated drives...
==> master: Forcing shutdown of VM...
==> master: Destroying VM and associated drives...
Done
-
Log in to the master as:
vagrant ssh master Last login: Wed Jul 15 20:36:32 2015 from 10.0.2.2 [vagrant@kubernetes-master ~]$
-
Log in as root:
[vagrant@kubernetes-master ~]$ su - Password: [root@kubernetes-master ~]#
Default root password for VM images created by Vagrant is ‘vagrant’.
-
Check the containers running on master:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dc59a764953c gcr.io/google_containers/etcd:2.0.12 "/bin/sh -c '/usr/lo 20 hours ago Up 20 hours k8s_etcd-container.fa2ab1d9_etcd-server-kubernetes-master_default_7b64ecafde589b94a342982699601a19_2b69c4d5 b722e22d3ddb gcr.io/google_containers/kube-scheduler:d1107ff3b8fcdcbf5a9d78d9d6dbafb1 "/bin/sh -c '/usr/lo 20 hours ago Up 20 hours k8s_kube-scheduler.7501c229_kube-scheduler-kubernetes-master_default_98b354f725c1589ea5a12119795546ae_b81b9740 38a73e342866 gcr.io/google_containers/kube-controller-manager:fafaf8100ccc963e643b55e35386d713 "/bin/sh -c '/usr/lo 20 hours ago Up 20 hours k8s_kube-controller-manager.db050993_kube-controller-manager-kubernetes-master_default_f5c25224fbfb2de87e1e5c35e6b3a293_dcd4cb5d 01001de6409e gcr.io/google_containers/kube-apiserver:cff9e185796caa8b281e7d961aea828b "/bin/sh -c '/usr/lo 20 hours ago Up 20 hours k8s_kube-apiserver.7e06f4e1_kube-apiserver-kubernetes-master_default_829f8c23fd5fc7951253cac7618447fc_b39c0a5d 0f8ccb144ece gcr.io/google_containers/pause:0.8.0 "/pause" 20 hours ago Up 20 hours k8s_POD.e4cc795_kube-scheduler-kubernetes-master_default_98b354f725c1589ea5a12119795546ae_eb1efcac 0b8f527456c0 gcr.io/google_containers/pause:0.8.0 "/pause" 20 hours ago Up 20 hours k8s_POD.e4cc795_kube-apiserver-kubernetes-master_default_829f8c23fd5fc7951253cac7618447fc_5dd4dee7 39d9c41ab1a2 gcr.io/google_containers/pause:0.8.0 "/pause" 20 hours ago Up 20 hours k8s_POD.e4cc795_kube-controller-manager-kubernetes-master_default_f5c25224fbfb2de87e1e5c35e6b3a293_522972ae d970ddff7046 gcr.io/google_containers/pause:0.8.0 "/pause" 20 hours ago Up 20 hours k8s_POD.e4cc795_etcd-server-kubernetes-master_default_7b64ecafde589b94a342982699601a19_fa75b27f