cf-ops-automation-broker

On demand dedicated services through COA concourse pipeline engine

Overview

COAB is a generic service broker which enables operators to provide on-demand dedicated services from existing bosh releases with service brokers offering shared service plans (e.g cassandra-cf-service-boshrelease)

COAB leverages concourse-based pipelines to deploy and operate the dedicated resources, managed by the cf-ops-automation (COA) collaboration framework.

Here are some slides to provide more background to this project:

The following diagram illustrates COAB interactions

1- A user requests a dedicated service instance (or binding) through its prefered platform (CF or K8S service catalog) which in turn propagate the request to COAB as an OSB API call
2- COAB requests an on-demand dedicated pipeline to the COA templating engine, by writing to Git repos
3- COA templating engine reads the git repos
4- COA generates on-demand pipelines that deploys and operate the requested dedicate service resources

5- The dedicated concourse pipeline provisions resources supporting the dedicated resource (in form of a bosh deployment, a terraform module, or a CF app)
7- the dedicated concourse pipeline records the outcome of the dedicated service in git
8- the dedicated concourse pipeline (and its underlying tools such as bosh director) records the credentials necessary to consumme the decicated service instance/binding
9- COAB pulls the dedicated service provisionning completion status from git
10a)- COAB delegates OSB API calls to nested service brokers, delegating credentials management to them
10b)- alternatively COAB fetches dedicated service credentials generated by the pipeline from credhub
11- COAB returns service instance/bindings to user

Status

This project is still in a beta state providing POC broker releases for CloudFlare and Cassandra use-cases above. Next step is to generalize the approach.

Operating COAB

Deploy the broker as a CF app:

• The CF manifest.yml file would be available in orange-cloudfoundry/paas-templates repo
  • The documentation reference for supported and required environment variables is in the integration tests properties.
  • The broker does not require stateful data service to be bound.

Configuring the service broker catalog

Use spring.cloud.openservicebroker.catalog environment variable to set catalog config in a YAML format. See spring-cloud-open-service-broker for a sample YML and raw properties configuration

Git clone pooling

By default coab perform pooling of git clone, including eager pooling at startup (i.e. prefetching a clone), and asynchronously maintaining the min idle pool size. This is designed to avoid getting the git clone elapsed time as part of the user-facing experience and running on the OSB sync timeout (which defaults to 60s)

Pooling may be disabled, through GitProperties.java and eager pool configured in PoolingProperties.java.

Read-only mode

When the service instance read-only mode is configured (see DeploymentProperties.java), then service instance operations (create/update/delete) are rejected, while service binding operations (create/delete) are still accepted. This enables to perform maintenance on the underlying COA/git branches infrastructure while not risking corrupted COA inventory and not imposing full control plan downtime to coab users.

skipDeProvision opt-in

When pipeline.skipDeProvision is set to true, then the inner broker will not receive unprovision calls (i.e. upon cf delete-service-instance). Default is false

This is designed to support undeletes by operators until they explicitly approve the deletion the associated COA deployment (and the associated underlying bosh deployment).

Some brokers with highly sensitive data (e.g. implementing data erasure) should prefer to not opt-in for this flag, as this could increase risk of data recovery after the service deprovision phase.

Disable input validation as a transient workaround

Coab strives to perform strict fail-fast input validation to prevent security-related injections in paas-templates.

When the input validation is too restrictive and blocks legitimate use-case, it is possible to transiently disable input validation through setting the deployment.coabVarsInputValidationDisabled property to true (default is false)

Troubleshooting COAB

Spring boot debug mode

Enable springboot debug/trace mode to diagnose beans configuration. When running COAB as a cf app, use the commands below to turn on spring boot debug mode.

cf set-env my-app DEBUG true

and when finished

cf unset-env my-app DEBUG true

Increase logging levels

Increase verbosity for COAB classes:

• default levels are in cf-ops-automation-bosh-broker/src/main/resources/application.yml
• see some frequently used levels in cf-ops-automation-bosh-broker/src/test/resources/application.properties and cf-ops-automation-broker-core/src/test/resources/application.yml

Expose more spring actuator endpoints

By default, COAB only exposed the health and info endpoints and make them accessible without authentication.

For debug purposes only, expose all actuator endpoints which are disabled by default, see related spring manual They currently require broker basic auth.

management.endpoints.web.exposure.include: "*"

See also, associated actuator endpoints API doc

Authoring a new COAB-based service

Prepare a standard COA bosh deployment, see COA documentation

such as

$ tree coab-depls/mongodb

coab-depls/mongodb
|-- deployment-dependencies.yml
`-- template
    |-- coab-operators.yml
    |-- add-prometheus-operators.yml
    |-- add-shield-operators.yml
    |-- mongodb-vars.yml
    `-- mongodb.yml
    
    
$ cat coab-depls/mongodb/template/coab-operators.yml 

#replace deployment name
- type: replace
  path: /name
  value: ((deployment_name))

#replace external host
- type: replace
  path: /instance_groups/name=mongodb-broker/jobs/name=route-registrar/properties/route_registrar/external_host
  # deployment name is of format model-deployment-short-alias + "_" + <service-instance-id>
  value: mongodb-broker-((deployment_name)).((!/secrets/cloudfoundry_system_domain))

COAB would generate the following for service instance provisionning/updates:

$tree coab-depls/m_f49911f7-b69a-4aba-afdf-23fd11014278

coab-depls/m_f49911f7-b69a-4aba-afdf-23fd11014278
|-- deployment-dependencies.yml -> ../../template/mongodb/deployment-dependencies.yml
`-- template
    |-- coab-operators.yml -> ../../mongodb/operators/coab-operators.yml
    |-- mongodb-vars.yml -> ../../mongodb/template/mongodb-vars.yml
    |-- m_f49911f7-b69a-4aba-afdf-23fd11014278.yml -> ../../mongodb/template/mongodb.yml
    `-- coab-vars.yml
    

The coab-vars.yml file contains the open service broker api standardized input for both service provisionning and update. The following example below shows a dummy example with a user passing arbirary params in a cf update-service-instance along with a service plan update.

$cat coab-depls/m_f49911f7-b69a-4aba-afdf-23fd11014278/coab-vars.yml

---
deployment_name: "m_f49911f7-b69a-4aba-afdf-23fd11014278"
instance_id: "f49911f7-b69a-4aba-afdf-23fd11014278"
service_id: "mongodb-service_definition_id"
plan_id: "mongodb-plan_guid"
context:
  platform: "cloudfoundry"
  user_guid: "user_guid1"
  space_guid: "space_guid1"
  organization_guid: "org_guid1"
parameters:
  cacheRatio: 0.8642
  cacheSizeMb: 10
  slowQuery: false
previous_values:
  plan_id: "previous_plan_guid"     

Note: for security reasons, input validation is applied on the name and value of arbitrary params to prevent various injections (such as reading from credhub, spruce, file system injection, yml loading or reference expansions), and only so alphabetical characters, numbers are supported.

Contributing

See CONTRIBUTING.md document