Skip to content

Commit

Permalink
Alpha.2 prerelase fixes (#159)
Browse files Browse the repository at this point in the history 
* Multiple documentation fixes for 0.0.0.alpha.2
* validation docs

Co-authored-by: Roberto Polli <robipolli@gmail.com>
Co-authored-by: Kyle Hodgetts <kyle@kubeshop.io>
Co-authored-by: George Dobrovolsky <georgy.dobrovolsky@gmail.com>
  • Loading branch information
@Tarick @ioggstream @dobegor
4 people authored Jan 21, 2022
1 parent 29e94ec commit 3d51723
Show file tree
Hide file tree
Showing 12 changed files with 317 additions and 71 deletions.
3 changes: 3 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,9 @@ spec:
description: API Management facade for a very handy and free online HTTP tool.
version: '1.0'
x-kusk:
validation:
request:
enabled: true # enable automatic request validation using OpenAPI spec
upstream:
service:
name: httpbin
Expand Down
12 changes: 12 additions & 0 deletions docs/customresources/api.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ Refer to [extention](../extension.md) for the further information on how to add

The required field of API resource is spec.**spec** where `x-kusk`-enhanced OpenAPI file is supplied as an embedded string. You can generate it (and integrate into your CI) using [kgw](https://github.com/kubeshop/kgw) CLI tool.

The optional spec.**fleet** field specifies to what Envoy Fleet (Envoy Proxy instances with the exposing K8s Service) this configuration applies to.
fleet.**name** and fleet.**namespace** reference the deployed EnvoyFleet Custom Resource name and namespace.
You can deploy you API configuration in any namespace with any name and it will be applied to the specific Envoy Fleet.
If this option is missing, the autodetection will be performed to find the single deployed in the Kubernetes cluster fleet which is thus considered as the default fleet.
The deployed API custom resource will be changed to map to that fleet accordingly.
If there are multiple fleets deployed, the spec.**fleet** is required to specify in the manifest.

Once the resource manifest is deployed, Kusk Gateway Manager will use it to configure routing for Envoy Fleet.
Multiple resources can exist in different namespaces, all of them will be evaluated and the configuration merged on any update with these resource.
Trying to apply a resource that has conflicting routes with the existing resources (i.e. same HTTP method and path) would result in error.
Expand All @@ -21,6 +28,11 @@ kind: API
metadata:
name: api-sample
spec:
# Envoy Fleet (its name and namespace) to deploy the configuration to, here - deployed EnvoyFleet with the name "default" in the namespace "default".
# Optional, if not specified - single (default) fleet autodetection will be performed in the cluster.
fleet:
name: default
namespace: default
# OpenAPI file with x-kusk annotation here
spec: |
openapi: 3.0.2
Expand Down
157 changes: 149 additions & 8 deletions docs/customresources/envoyfleet.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,170 @@

This resource defines the EnvoyFleet, which is the implementation of the gateway in Kubernetes based on Envoy Proxy.

Once the resource manifest is deployed to Kubernetes, it is used by Kusk Gateway Manager to setup K8s Envoy Proxy **Deployment** and **Service** with the type *LoadBalancer*.
Once the resource manifest is deployed to Kubernetes, it is used by Kusk Gateway Manager to setup K8s Envoy Proxy **Deployment**, **ConfigMap** and **Service**.

Envoy Proxy is configured to connect to the [XDS](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol) service of the KGW Manager to retrieve the configuration.
In its initial state there is no configuration, you have to deploy API or StaticRoute resource to setup the routing.
The **ConfigMap** config bootstraps Envoy Proxy to connect to the [XDS](https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol) service of the KGW Manager to retrieve the configuration.
In its initial state there is a minimal configuration, you have to deploy API or StaticRoute resource to setup the routing.

If the Custom Resource is uninstalled, the Manager deletes the created K8s resources.

Currently supported parameters:
You can deploy multiple Envoy Fleets and thus have multiple Gateways available.

* spec.**size** , which is the number of Envoy Proxy pods in the K8s deployment, defaults to 1 if not specified.
* metadata.**name** is the EnvoyFleet ID. The Manager will supply the configuration for this specific ID, matching the API / StaticRoute by fleet name.
Once the fleet is deployed, it **status** field shows the success of the process (Deployed, Failed), so it can be shown with ```kubectl describe envoyfleet``` command.

**Alpha Limitations**:

* the support for multiple IDs configuration is a work in progress, right now the static ID **default** is used.
* currently it shows only the success of K8s resources deployment, it doesn't show if the Envoy Proxy pods are alive and if the Service has the External IP Address allocated.

Currently supported parameters:

* metadata.**name** and metadata.**namespace** are used as the EnvoyFleet ID. The Manager will supply the configuration for this specific ID - Envoy will connect to the KGW Manager with it. API/StaticRoute can be deployed to this fleet using their fleet name field.

* spec.**image** is the Envoy Proxy container image tag, usually envoyproxy/envoy-alpine.

* spec.**service** defines the configuration of the K8s Service that exposes Envoy Proxy deployment. It has similar to the K8s Service configuration but with the limited set of fields.
* spec.service.**type** - select the Service type (NodePort, ClusterIP, LoadBalancer).
* spec.service.**ports** - expose TCP ports (80, 443 or any other), routed to the ports names that Deployment exposes (http, https) - ports that Envoy Proxy listener binds to.
* spec.service.**annotations** - add annotations to the Service that will control load balancer configuration in the specific cloud providers implementations (e.g. to setup the internal Google Cloud Platform load balancer in Google Cloud Engine we annotate Service with the related annotation).
* spec.service.**loadBalancerIP** can be used to specify the pre-allocated Load Balancer IP address, so it won't be deleted in case the Service is deleted.
* spec.service.**externalTrafficPolicy** optional parameter denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. "Local" preserves the client source IP and avoids a second hop for LoadBalancer and Nodeport type services, but risks potentially imbalanced traffic spreading. "Cluster" obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading. For the preservation of the real client ip in access logs chose "Local"

* spec.**size** optional parameter is the number of Envoy Proxy pods in the K8s deployment, defaults to 1 if not specified.

* spec.**resources** optional parameter configures Envoy Proxy container CPU/Memory requests and limits - read [Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for the details. By default we don't specify any requests or limits.

* spec.**annotations** optional parameter adds additional annotations to the Envoy Proxy pods, e.g. for the Prometheus scraping.

* spec.**nodeSelector**, spec.**tolerations** and spec.**affinity** optional parameters provide the Envoy Proxy deployment settings for the K8s Pod scheduler. Read [Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) to understand how can you bind Envoy pods to some type of nodes (e.g. non-preemtible node type pool) and how to ensure that Envoy pods are placed onto the different nodes for the High Availability. See the YAML example below too. The structure of these fields are the same as for K8s Deployment. All these options could be used simultaneously influencing each other.

* spec.**accesslog** optional parameter defines Envoy Proxy stdout HTTP requests logging. Each Envoy pod can stream access log to stdout. If not specified - no streaming. If specified, you must chose the **format** and optionally - text or json template to tune the output.
* spec.accesslog.**format** required parameter specifies the format of the output **json** (structured) or **text**. Note that json format doesn't preserve fields order.
* spec.accesslog.**text_template**|**json_template** optional parameters could be used to specify what exactly available Envoy request data to log. See [Envoy's Access Logging](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-strings) for the details. If not specified any - use Kusk Gateway provided defaults.

* spec.**tls** optional parameter defines TLS settings for the EnvoyFleet. If not specified, the EnvoyFleet will accept only HTTP traffic.

* currently resource **status** field is not updated by the manager when the reconsiliation of the configuration finishes.
* spec.tls.**cipherSuites** An optional field, if specified, the TLS listener will only support the specified cipher list when negotiating TLS 1.0-1.2 (this setting has no effect when negotiating TLS 1.3). If not specified, a default list will be used. Defaults are different for server (downstream) and client (upstream) TLS configurations. For more information see: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/transport_sockets/tls/v3/common.proto

* spec.tls.**tlsMinimumProtocolVersion** An optional field specifying the minimum TLS protocol version. By default, it’s TLSv1_2 for clients and TLSv1_0 for servers.

* spec.tls.**tlsMaximumProtocolVersion** An optional field specifying the maximum TLS protocol version. By default, it’s TLSv1_2 for clients and TLSv1_3 for servers.

* spec.tls.**tlsSecrets** Secret name and namespace combinations for locating TLS secrets containing TLS certificates.
You can specify more than one.
Kusk Gateway Manager pulls the certificates from the secrets, extracts the matching hostnames from SubjectAlternativeNames (SAN) certificate field and configures Envoy to use that certificate for those hostnames.

* spec.tls.tlsSecrets.**secretRef** is the name of the Kubernetes secret containing the TLS certificate.

* spec.tls.tlsSecrets.**namespace** is the namespace where that Kubernetes secret resides.

```yaml EnvoyFleet.yaml
apiVersion: gateway.kusk.io/v1alpha1
kind: EnvoyFleet
metadata:
name: default
namespace: default
spec:
image: "envoyproxy/envoy-alpine:v1.20.0"
service:
# NodePort, ClusterIP, LoadBalancer
type: LoadBalancer
# Specify annotations to modify service behaviour, e.g. for GCP to create internal load balancer:
# annotations:
# networking.gke.io/load-balancer-type: "Internal"
# Specify preallocated static load balancer IP address if present
#loadBalancerIP: 10.10.10.10
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: http
# externalTrafficPolicy: Cluster|Local
resources:
# limits:
# cpu: 1
# memory: 100M
requests:
cpu: 10m
memory: 100M
# Put any additional annotations to the Enovy pod.
# Here we add the annotations for the Prometheus service discovery to scrape Envoy pods for the Prometheus metrics.
# annotations:
# prometheus.io/scrape: 'true'
# prometheus.io/port: '19000'
# prometheus.io/path: /stats/prometheus

##### Scheduling directives
# Read https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ for the details.

# Optional - schedule Envoy pods to the node with the label "disktype=ssd".
# nodeSelector:
# disktype: "ssd"

# Optional - allow to be scheduled on the "tainted" node. Taint with "kubectl taint nodes node1 key1=value1:NoSchedule".
# Taints will repel the pods from the node unless the pods have the specific toleration.
# The line below will allow this specific Envoy pod to be scheduled there (but scheduler decideds where to put it anyway).
# tolerations:
# - key: "key1"
# operator: "Exists"
# effect: "NoSchedule"

# Optional - provide pods affinity and anti-affinity settings.
# This is more flexible than nodeSelector scheme but they can be specified together.
# For the scalability and fault tolerance we prefer to put all Envoy pods onto different nodes - in a case one node fails we survive on others.
# The block below will search for all matching labels of THIS "default" envoy fleet pods and will try to schedule pods onto different nodes.
# Other fleets (if present) are not taken into consideration. You can specify nodeAffinity and podAffinity as well.
# affinity:
# podAntiAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# - labelSelector:
# matchExpressions:
# - key: app.kubernetes.io/name
# operator: In
# values:
# - kusk-gateway-envoy-fleet
# - key: fleet
# operator: In
# values:
# - default
# topologyKey: kubernetes.io/hostname

# optional, the number of Envoy Proxy pods to start
size: 1

# Access logging to stdout
# Optional, if this is missing no access logging to stdout will be done
accesslog:
# json|text
format: text
# Depending on format we can specify our own log template or if template is not specified - default Kusk Gateway will be used.
# See https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-strings for the details.
# Below are specified the examples of similar and minimalistic formats for both text and json format types.
# Text format fields order is preserved.
# The output example:
# "[2021-12-15T16:50:50.217Z]" "GET" "/" "200" "1"
text_template: |
"[%START_TIME%]" "%REQ(:METHOD)%" "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" "%RESPONSE_CODE%" "%DURATION%"
# Json format fields order isn't preserved
# The output example:
# {"start_time":"2021-12-15T16:46:52.135Z","path":"/","response_code":200,"method":"GET","duration":1}
json_template:
start_time: "%START_TIME%"
method: "%REQ(:METHOD)%"
path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
response_code: "%RESPONSE_CODE%"
duration: "%DURATION%"

# TLS configuration
# tls:
# cipherSuites:
# - ECDHE-ECDSA-AES128-SHA
# - ECDHE-RSA-AES128-SHA
# - AES128-GCM-SHA256
# tlsMinimumProtocolVersion: TLSv1_2
# tlsMaximumProtocolVersion: TLSv1_3
# tlsSecrets:
# - secretRef: my-cert
# namespace: default
```
23 changes: 21 additions & 2 deletions docs/customresources/staticroute.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,11 @@ kind: StaticRoute
metadata:
name: staticroute-sample
spec:
# Envoy Fleet (its name and namespace) to deploy the configuration to, here - deployed EnvoyFleet with the name "default" in the namespace "default".
# Optional, if not specified - single (default) fleet autodetection will be performed in the cluster.
fleet:
name: default
namespace: default
hosts: [<string>, <string>, ...]
paths:
# Consists of path matchers with HTTP methods (lowercase), which in turn either:
Expand Down Expand Up @@ -91,6 +96,8 @@ spec:
- X-Custom-Header2
# how long to cache this CORS information for the browser, returned with Access-Control-Max-Age.
max_age: <int>
# Enable establishing Websockets connections
websocket: <true|false>
# "redirect" creates HTTP redirect to other endpoint. Mutually exclusive with "route".
redirect:
# redirect to http or https
Expand All @@ -114,6 +121,16 @@ spec:
-- skipped --
```

## Envoy Fleet

spec.**fleet** optional field specifies to what Envoy Fleet (Envoy Proxy instances with the exposing K8s Service) this configuration applies to.
fleet.**name** and fleet.**namespace** reference the deployed EnvoyFleet Custom Resource name and namespace.

You can deploy you StaticRoute configuration in any namespace with any name and it will be applied to the specific Envoy Fleet.
If this option is missing, the autodetection will be performed to find the single deployed in the Kubernetes cluster fleet which is thus considered as the default fleet.
The deployed StaticRoute custom resource will be changed to map to that fleet accordingly.
If there are multiple fleets deployed, the spec.**fleet** is required to specify in the manifest.

## Request matching

We match the incoming request by HOST header, path and HTTP method.
Expand Down Expand Up @@ -239,10 +256,11 @@ route:
- X-Custom-Header2
# how long to cache this CORS information for the browser in seconds, returned with Access-Control-Max-Age header
max_age: <int>
# Enable establishing Websockets connections
websocket: <true|false>

```


*route*.**upstream** is a required field that defines the upstream host parameters.
We proxy using DNS hostname or local cluster K8s service parameters, which are further resolved to DNS hostname. Either *upstream*.**host** or *upstream*.**service** must be specified inside.

Expand All @@ -251,9 +269,9 @@ We proxy using DNS hostname or local cluster K8s service parameters, which are f
*route*.**qos** optional field is the container for request Quality of Service parameters, i.e. timeouts, failure retry policy.

*route*.**cors** optional field is the container for [Cross-Origin Resource Sharing](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) headers parameters. If this field is specified, route will be augmented with CORS preflight OPTIONS HTTP method matching. This will allow Envoy to return the response to OPTIONS request with the specified here CORS headers to the user without proxying to upstream. It is advised to read [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) before trying to configure this.

Note: the structure for CORS specified above is an example, i.e. one should write its own set of methods and headers.

*route*.**websocket** optional boolean field defines whether to enable handling of "Upgrade: websocket" and related Websocket HTTP headers in the request to create a Websocket tunnel to the backend. By default false, don't handle Websockets.

## Example

Expand Down Expand Up @@ -350,6 +368,7 @@ spec:
expose_headers:
- X-API-VERSION
max_age: 8600
websocket: true
post: *old_api_route
put: *old_api_route
patch: *old_api_route
Expand Down
Loading

0 comments on commit 3d51723

Please sign in to comment.