Skip to content

Commit

Permalink
blue/green deployment
Browse files Browse the repository at this point in the history
  • Loading branch information
jcmoraisjr committed Apr 22, 2018
1 parent 6c3dc35 commit 90e858a
Show file tree
Hide file tree
Showing 11 changed files with 370 additions and 18 deletions.
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ The following annotations are supported:
|`[0]`|[`ingress.kubernetes.io/auth-tls-cert-header`](#auth-tls)|[true\|false]|[doc](/examples/auth/client-certs)|
||[`ingress.kubernetes.io/auth-tls-error-page`](#auth-tls)|url|[doc](/examples/auth/client-certs)|
||[`ingress.kubernetes.io/auth-tls-secret`](#auth-tls)|namespace/secret name|[doc](/examples/auth/client-certs)|
|`[1]`|[`ingress.kubernetes.io/blue-green-deploy`](#blue-green)|label=value=weight,...|[doc](/examples/blue-green)|
|`[0]`|[`ingress.kubernetes.io/hsts`](#hsts)|[true\|false]|-|
|`[0]`|[`ingress.kubernetes.io/hsts-include-subdomains`](#hsts)|[true\|false]|-|
|`[0]`|[`ingress.kubernetes.io/hsts-max-age`](#hsts)|qty of seconds|-|
Expand Down Expand Up @@ -113,6 +114,34 @@ The following annotations are supported:

See also client cert [sample](/examples/auth/client-certs).

### Blue-green

Configure weight of a blue/green deployment. The annotation accepts a comma separated list of label
name/value pair and a numeric weight. Concatenate label name, label value and weight with an equal
sign, without spaces. The label name/value pair will be used to match corresponding pods.

The endpoints of a single backend are selected using service selectors, which also uses labels.
Because of that, in order to use blue/green deployment, the deployment, daemon set or replication
controller template should have at least two label name/value pairs - one that matches the service
selector and another that matches the blue/green selector.

The following configuration `group=blue=1,group=green=4` will redirect 20% of the load to the
`group=blue` pods and 80% of the load to the `group=green` if they have the same number of replicas.

Note that this configuration is related to every single pod. On the configuration above, if
`group=blue` has two replicas and `group=green` has just one, green would receive only the double
of the number of requests dedicated to blue. This can be adjusted using higher numbers - eg `10/40`
instead of `1/4` - and divided by the number of replicas of each deployment - eg `5/40` instead of
`10/40`.

Value of `0` (zero) can also be used. This will let the endpoint configured in the backend accepting
persistent connections - see [affinity](#affinity) - but will not participate in the load balancing.
The maximum weight value is `256`.

See also the [example](/examples/blue-green) page.

http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-weight

### Limit

Configure rate limit and concurrent connections per client IP address in order to mitigate DDoS attack.
Expand Down
114 changes: 114 additions & 0 deletions examples/blue-green/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# HAProxy Ingress blue/green deployment

This example demonstrates how to configure
[blue/green deployment](https://www.martinfowler.com/bliki/BlueGreenDeployment.html)
on HAProxy Ingress controller.

## Prerequisites

This document has the following prerequisite:

* A Kubernetes cluster with a running HAProxy Ingress controller. See the [five minutes deployment](/examples/setup-cluster.md#five-minutes-deployment) or the [deployment example](/examples/deployment)

## Deploying applications

In order to the configuration have effect, at least two deployments, or daemon sets, or replication
controllers should be used with at least two pairs of label name/value.

The following instructions create two deployment objects using `run` label as the service selector
and `group` label as the blue/green deployment selector:

```
$ kubectl run blue \
--image=gcr.io/google_containers/echoserver:1.3 \
--port=8080 --labels=run=bluegreen,group=blue
deployment "blue" created
$ kubectl run green \
--image=gcr.io/google_containers/echoserver:1.3 \
--port=8080 --labels=run=bluegreen,group=green
deployment "green" created
```

Certify if the pods are running and have the correct labels. Note that both `group` and `run`
labels were applied:

```
$ kubectl get pod -lrun=bluegreen --show-labels
NAME READY STATUS RESTARTS AGE LABELS
blue-79c9b67d5b-5hd2r 1/1 Running 0 35s group=blue,pod-template-hash=3575623816,run=bluegreen
green-7546d648c4-p7pmz 1/1 Running 0 28s group=green,pod-template-hash=3102820470,run=bluegreen
```

# Configure

Create a service that bind both deployments together using the `run` label. The expose command need
a deployment object, take anyone, we will override it's selector:

```
$ kubectl expose deploy blue --name bluegreen --selector=run=bluegreen
service "bluegreen" exposed
$ kubectl get svc bluegreen -otemplate --template '{{.spec.selector}}'
map[run:bluegreen]
```

Check also the endpoints, it should list both blue and green pods:

```
$ kubectl get ep bluegreen
NAME ENDPOINTS AGE
bluegreen 172.17.0.11:8080,172.17.0.19:8080 2m
$ kubectl get pod -lrun=bluegreen -owide
NAME READY STATUS RESTARTS AGE IP NODE
blue-79c9b67d5b-5hd2r 1/1 Running 0 2m 172.17.0.11 192.168.100.99
green-7546d648c4-p7pmz 1/1 Running 0 2m 172.17.0.19 192.168.100.99
```

Configure the ingress resource:

```
$ kubectl create -f - <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
ingress.kubernetes.io/blue-green-deploy: group=blue=1,group=green=4
name: bluegreen
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: bluegreen
servicePort: 8080
path: /
EOF
```

```
$ kubectl get ing
NAME HOSTS ADDRESS PORTS AGE
bluegreen example.com 80 11s
```

Lets test! Change the IP below to your HAProxy Ingress controller:

```
$ for i in `seq 1 50`; do
echo -n "."
curl -s 192.168.100.99 -H 'Host: example.com' >/dev/null
done
..................................................
```

Now check `<ingress-ip>:1936` - the stats page has a `<namespace>-bluegreen-8080` card
with backend stats. The column Session/Total should have about 20% of the deployments
on one endpoint and 80% on another.

The blue/green configuration is related to every single pod - if the number of replicas change,
the load will also change between blue and green groups. Have a look at the
[doc](/README.md#blue-green) on how it works.
115 changes: 115 additions & 0 deletions pkg/common/ingress/annotations/bluegreen/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
/*
Copyright 2018 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package bluegreen

import (
"fmt"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/parser"
extensions "k8s.io/api/extensions/v1beta1"
"strconv"
"strings"
)

const (
blueGreenAnn = "ingress.kubernetes.io/blue-green-deploy"
)

// DeployWeight has one label name/value pair and it's weight
type DeployWeight struct {
LabelName string
LabelValue string
Weight int
}

// Config is the blue/green deployment configuration
type Config struct {
DeployWeight []DeployWeight
}

type bgdeploy struct {
}

// NewParser creates a new blue/green annotation parser
func NewParser() parser.IngressAnnotation {
return bgdeploy{}
}

// Parse parses blue/green annotation and create a Config struct
func (bg bgdeploy) Parse(ing *extensions.Ingress) (interface{}, error) {
s, err := parser.GetStringAnnotation(blueGreenAnn, ing)
if err != nil {
return nil, err
}
weights := strings.Split(s, ",")
var dw []DeployWeight
for _, weight := range weights {
dwSlice := strings.Split(weight, "=")
if len(dwSlice) != 3 {
return nil, fmt.Errorf("invalid weight format on blue/green config: %v", weight)
}
w, err := strconv.ParseInt(dwSlice[2], 10, 0)
if err != nil {
return nil, fmt.Errorf("error reading blue/green config: %v", err)
}
if w < 0 {
w = 0
}
dwItem := DeployWeight{
LabelName: dwSlice[0],
LabelValue: dwSlice[1],
Weight: int(w),
}
dw = append(dw, dwItem)
}
return &Config{
DeployWeight: dw,
}, nil
}

// Equal tests equality between two Config objects
func (b1 *Config) Equal(b2 *Config) bool {
if len(b1.DeployWeight) != len(b2.DeployWeight) {
return false
}
for _, dw1 := range b1.DeployWeight {
found := false
for _, dw2 := range b2.DeployWeight {
if (&dw1).Equal(&dw2) {
found = true
break
}
}
if !found {
return false
}
}
return true
}

// Equal tests equality between two DeployWeight objects
func (dw1 *DeployWeight) Equal(dw2 *DeployWeight) bool {
if dw1.LabelName != dw2.LabelName {
return false
}
if dw1.LabelValue != dw2.LabelValue {
return false
}
if dw1.Weight != dw2.Weight {
return false
}
return true
}
13 changes: 13 additions & 0 deletions pkg/common/ingress/controller/annotations.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/auth"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/authreq"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/authtls"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/bluegreen"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/clientbodybuffersize"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/cors"
"github.com/jcmoraisjr/haproxy-ingress/pkg/common/ingress/annotations/defaultbackend"
Expand Down Expand Up @@ -81,6 +82,7 @@ func newAnnotationExtractor(cfg extractorConfig) annotationExtractor {
"SecureUpstream": secureupstream.NewParser(cfg, cfg),
"ServiceUpstream": serviceupstream.NewParser(),
"SessionAffinity": sessionaffinity.NewParser(),
"BlueGreen": bluegreen.NewParser(),
"SSLPassthrough": sslpassthrough.NewParser(),
"ConfigurationSnippet": snippet.NewParser(),
"Alias": alias.NewParser(),
Expand Down Expand Up @@ -130,6 +132,7 @@ func (e *annotationExtractor) Extract(ing *extensions.Ingress) map[string]interf
const (
secureUpstream = "SecureUpstream"
healthCheck = "HealthCheck"
blueGreen = "BlueGreen"
sslPassthrough = "SSLPassthrough"
sessionAffinity = "SessionAffinity"
serviceUpstream = "ServiceUpstream"
Expand Down Expand Up @@ -160,6 +163,16 @@ func (e *annotationExtractor) HealthCheck(ing *extensions.Ingress) *healthcheck.
return val.(*healthcheck.Upstream)
}

func (e *annotationExtractor) BlueGreen(ing *extensions.Ingress) *bluegreen.Config {
val, err := e.annotations[blueGreen].Parse(ing)
if err != nil {
return &bluegreen.Config{
DeployWeight: []bluegreen.DeployWeight{},
}
}
return val.(*bluegreen.Config)
}

func (e *annotationExtractor) SSLPassthrough(ing *extensions.Ingress) bool {
val, _ := e.annotations[sslPassthrough].Parse(ing)
return val.(bool)
Expand Down
52 changes: 50 additions & 2 deletions pkg/common/ingress/controller/controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,7 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress)

for _, ing := range ingresses {
affinity := ic.annotations.SessionAffinity(ing)
blueGreen := ic.annotations.BlueGreen(ing)
anns := ic.annotations.Extract(ing)

for _, rule := range ing.Spec.Rules {
Expand Down Expand Up @@ -596,12 +597,14 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress)

locs[host] = append(locs[host], path.Path)
}

if len(ups.BlueGreen.DeployWeight) == 0 {
ups.BlueGreen = *blueGreen
}
}
}
}

aUpstreams := make([]*ingress.Backend, 0, len(upstreams))

for _, upstream := range upstreams {
var isHTTPSfrom []*ingress.Server
for _, server := range servers {
Expand All @@ -628,6 +631,51 @@ func (ic *GenericController) getBackendServers(ingresses []*extensions.Ingress)
}
}

// calc deployment weight of a blue/green deployment
for _, upstream := range upstreams {
svc := upstream.Service
podNamespace := svc.Namespace
deployWeight := upstream.BlueGreen.DeployWeight
for epID := range upstream.Endpoints {
ep := &upstream.Endpoints[epID]
if ep.Target == nil {
glog.Warningf("ignoring blue/green config due to empty object reference on endpoint %v/%v", podNamespace, upstream.Name)
continue
}
if len(deployWeight) == 0 {
// no blue/green config, removing weight config and skipping to the next
ep.Weight = -1
continue
}
podName := ep.Target.Name
weight := -1
if pod, err := ic.listers.Pod.GetPod(podNamespace, podName); err == nil {
for _, weightConfig := range deployWeight {
if label, found := pod.Labels[weightConfig.LabelName]; found {
if label == weightConfig.LabelValue {
if weight < 0 {
weight = weightConfig.Weight
} else if weightConfig.Weight != weight {
glog.Warningf("deployment weight %v to service %v/%v is duplicated and was ignored", weightConfig.Weight, podNamespace, svc.Name)
}
}
} else {
glog.Warningf("pod %v/%v does not have label %v used on blue/green deployment", podNamespace, podName, weightConfig.LabelName)
}
}
} else {
glog.Warningf("could not calc weight of pod %v/%v: %v", podNamespace, podName, err)
}
// weight wasn't assigned, set as zero to remove all the traffic
// without removing from the balancer
if weight < 0 {
weight = 0
}
ep.Weight = weight
}
}

aUpstreams := make([]*ingress.Backend, 0, len(upstreams))
// create the list of upstreams and skip those without endpoints
for _, upstream := range upstreams {
if len(upstream.Endpoints) == 0 {
Expand Down
Loading

0 comments on commit 90e858a

Please sign in to comment.