Automation Evolution The Path to Intelligent Orchestration
Orchestration can be viewed as part of an automation continuum. Of course, all software is automation. That's the essence of it. Layer upon layer of abstraction has been built to improve productivity and understanding of complex systems. The cloud revolution is built up these abstractions, going beyond the abstraction of mere application logic, to the computer itself. These abstractions are the building blocks of the modern computing world, expressed as APIs. Of course there is nothing new or remarkable about APIs. They represents programming as usual, just with ever more powerful function calls. Declarative programming seeks to take the next step to make system abstractions more tangible, while hiding implementation detail and delivering ever greater leverage in the goal for constructing complex yet comprehensible systems. This article examines the industry trends towards declarative programming, and its key role in the future of intelligent orchestration.
Declarative programming provides yet another layer of abstraction beyond APIs where the initial state of a system is specified, and the implementation is not. By "implementation", traditional procedural/imperative logic is meant. This is hardly remarkable in computing, after all any function you might call in a programming language implies an action (add 2 numbers, for example), without specifying the implementation (which ultimately is done by applying voltages to logic circuits). You won't find "if" statements or loops in declarative languages, since those are imperative in nature. The typically examples of declarative programming such as SQL, HTML, even Kubernetes, have a domain limitation. In other words, they aren't general purpose programming languages; they are targeted at a particular problem domain such as relational databases, user interfaces, or the Kubernetes platform. Without such limitations/assumptions, the declarative language would lose it's power. It's an old lesson from the industrial revolution: constraining a problem space to a limited set up components enables unheard of levels of productivity and then affluence. The key lesson is parts standardization, which in the software era can be finely tailored to each use case.
The enemy of art is the absence of limitations - Orson Welles
So why is that important? Declarative languages can bridge the gap between programmers and non-programmers, which can make system design more accessible and understandable. The "declaration" (sometimes called model or intent) of the designer is interpreted by an imperative engine specific to the problem domain. In the case of SQL, a query planner evaluates SQL and produces a database specific set of commands. In the case HTML, the browser engine produces a series of drawing commands, and supplies events. Likewise, Kubernetes resource descriptors are interpreted by controllers (e.g. the scheduler) to perform complex operations in the Kubernetes environment.
Declarative languages shine in organizations where a "designer" role, tied to a platform with a limited number of options, are paired. The declarative vocabulary of the target environment can be tailored for the particular technology stack and business case. In these kinds of environments, the palette of options can be constrained, and the repetitive work of service orchestration can be automated. A concrete use case is a telecom service provider, which has a defined technology stack, and must perform service onboarding on a regular basis. So how is this symbolic vocabulary defined?
TOSCA2 is an OASIS project that aspires to "enhance the portability and operation of applications". TOSCA defines a "blueprint" resource that describes a topology in the application orchestration domain. The blueprint is the container for a model that describes the application to be orchestrated. The model is supplied to an "orchestrator", which is not defined by TOSCA. The blueprint itself specifies the orchestrated elements and their relationships using an extensible language that is object oriented in nature. The orchestrated elements consist of a vocabulary defined by TOSCA, hence it is a language for creating declarative domain specific vocabularies (hence "meta-language"). An example of the this is the TOSCA NFV Profile3. However, this language is far more general purpose than application modeling. The "TO" in TOSCA stands for "Topology Orchestration". This is no accident. TOSCA expresses the components of an orchestration as geometry, not logic. Defining orchestrations this was is far easier for people to understand, technical or not. It's the reason you sketch boxes and lines when brainstorming, and not pseudo-code.
People often say 'A picture is worth a thousand words.' I believe the original quote was actually 'A picture is worth ten thousand words' as stated by Fred R. Barnard, of Printers' Ink, 10 March 1927
A blueprint/model represents application elements (aka nodes) as "types", although a useful way to think of TOSCA types is as "classes" from the object oriented programming world. Types have properties and methods/operations. The methods are callbacks that the orchestrator calls via "workflows" (IOC1 pattern). You could think of operations as serving the same need as Javascript event handlers in a web page. Operations are grouped into "interfaces". Workflows typically look for nodes that implement interfaces that they are aware of, and call the related operations in an order dependent on the workflow purpose. For example, the "install" workflow in Cloudify looks for the "cloudify.interfaces.lifecycle" interface on node instances and calls operations that are defined there, which are specific to the node. The job of workflows is to build task trees which are optimally executed by the orchestrator.
The blueprint composes the building blocks or design vocabulary of the target environment in an orchestration. Each element in the blueprint has its own encapsulated logic that the orchestrator calls at the time appropriate for a certain workflow. Blueprints themselves are parameterized with inputs and outputs, and are turned into a living runtime model of the orchestrated system, which can respond to events and be manipulated by workflows.
Kubernetes has an opinionated view of infrastructure. In contrast, Cloudify is general purpose. However, there are some similarities. Both Kubernetes and Cloudify use a declarative approach to orchestration. Kubernetes has a concept of controllers which react to “resources”. Each resource type might have a different controller that keeps the definition in sync with the orchestrated system. Individual resources have no intrinsic knowledge of each other, and there is no real “blueprint” or “relationship” concept ( Helm is a partial attempt at such ). In Cloudify, orchestration processes are defined in blueprints and relationships are managed by the orchestrator. This allows the orchestrator have a “birds eye view” of the deployment, and also enables sophisticated day 2 operations called “workflows”. A workflow in Cloudify is a function that automates day 2 operations by having full access to the target deployment. Kubernetes has no such functionality because there is no concept internally of multi-resource orchestration. Because Cloudify has this view of the orchestration, it is able to automatically heal and scale infrastructure, whereas Kubernetes requires the underlying resources be coded for such. For example, in Cloudify, the orchestrator can update a load balancer automatically when a compute instance (or container) is added or removed. In Kubernetes, a special resource type must be used, and that resource type only supports the Kubernetes supplied load balancer. Other functionality such as rolling upgrades are supported in Kubernetes only among instances of the same container, whereas the workflow capability in Cloudify make such automation open ended and as complex as needed.
The point is not to point out the differences, but the similarities, driven by the same concerns. Model driven application orchestration leads to well defined systems with traditional imperative logic encapsulated (either in TOSCA workflows or Kubernetes operators).
There are three factors at play in orchestration automation: abstraction, intelligence and complexity. The first two are remedies for the last. If I have a very limited complexity (AKA choices or flexibility), I don't need much abstraction or intelligence in the system. This is just traditional rote programming to fixed requirements. If I want flexibility in a problem space with many choices, then I must pay for it with increased abstraction, intelligence, or complexity. Increasing abstraction is a traditional way to combat complexity, but has it's limits. Abstractions are approximations by nature, and typically hide complexity, but also functionality. They are, in a sense trading ignorance for simplicity. System intelligence lets us push back against functionality loss by being able to apply "intent" at a low level of abstraction. Think of intelligence in this context as a range from heuristics up to automated reasoning. A sufficiently intelligent system wouldn't even use abstractions. Intelligence delivers a different kind of abstraction, one that shortens the distance between requirements and implementation.
So how do we connect this to declarative orchestration? A model can be understood to be a goal state for a system. Currently our example models are very specific. Connect router to network. Connect instance to port. Even though I haven't specific how to instantiate or connect these in my model, they are still pretty low level. But what happens when we push the level of abstraction higher? That means relying more on the orchestrator to fill in the blanks. I may start with:
Imagine "Router X" here is a specific router model. The orchestrator has all information directly from the topology of the model. The next level of abstraction is just providing routing requirements, and let the orchestrator pick the best one base on them:
This is the idea behind requirements and capabilities in TOSCA. The orchestrator fulfills requirements, not specifications. It does this by selecting the best match from predefined types. Pay close attention to the terminology "requirements", as it indicates the direction of intelligent orchestration.
But we can go much further, depending on the intelligence of the orchestrator. The path is through the idea of requirements. As the requirements become higher level, the intelligence of the orchestrator must either increase, or it must be highly opinionated so that there is only very limited choice. Lets think about the requirements for an SD-WAN system. SD-WAN delivers flexibility and efficiency for WAN connections using software defined networking. Some example requirements: for live video applications, connect a central office and branch office via the highest bandwidth connection with highest consistency (MPLS), for voice use LTE, and for anything else use the internet. There are a ton of other detailed requirements in a real system, but we'll ignore those.
Given the requirements, the first question is: what vocabulary do I want to use to describe it? TOSCA can model entities at any level of abstraction, so you can really think hard about the requirements directly. I may have a team creating assembling such abstractions into blueprints frequently. What do they really need to know to do their job? First of all, there are a limited number of components available to accomplish the requirements. So the default approach is to create types to represent each one. This would give the orchestration designer the greatest flexibility, but also the highest complexity.
The designer has a full palette of options based on the available technology stack, and assembles the orchestration from them. This is sufficient for many real world scenarios. This is a simplified view to communicate a point, but imaging my stack consisted of dozens or hundreds of potential selections. We can crank up the type abstractions a bit more to yield more simplicity for the designer.
In this example, the individual types have been combined. This is not always possible, depending on the configuration attributes and needs of the implementation. Rather than selecting from a palette of different firewall vendors, the designer sees just one generic function, and plugs it in. The actual implementation can be selected by a policy, or by a requirements/capabilities match as in the earlier example. In practice we may need both levels of abstraction available, since there may be edge cases that require low level control. We can go further.
Now we're well down the abstraction rabbit hole. Instead of modeling individual components, I model sites and the relationships between them. Recall that each element has it's own configuration properties and lifecycle logic. In this example, it's the relationship itself that drives the orchestration. In TOSCA, relationships are first class entities in their own right, with no less power than types. What I've really done is cranked up the abstraction at the cost of flexibility. If there are few choices, or very well defined limited combinations of choices (perhaps blueprints in their own right), about how the underlying site connection is configured, then we won't get into trouble pursuing this route. There are advantages to pushing in this extreme direction:
- simplicity - can't get much simpler
- standardization - reducing choices reduces errors
- security - easier to vet simple orchestrations using certified components
Now consider a truly intelligent orchestrator. An intelligent orchestrator would consume the original requirements directly and emit the orchestration. It would have a concept of the target environment, the capabilities of the various networks, and the capabilities of the various other components. It would select MPLS because it best fit the requirement for video traffic, not because somebody modeled it. We're not there yet. But such a system could reason about the world if modeled in a declarative fashion. To reach this level of automation, not only the problem has to be modeled, but also the target environment. Such automation is not going to emerge from analyzing collections of shell scripts or other imperative artifacts.
The last contributor to intelligent orchestration discussed here is "policies". Policies refer to modes of operation and operational guidelines set prior to orchestration fulfillment. Policies very open ended, consisting of a named collection of properties. An orchestrator might pre-define generic policies for security, placement, scaling, cost etc. A custom policy might advise the router selection process above and limit the candidate list based on some arbitrary criteria, or it might limit access for a class of users when configuring the MPLS connection above. It might define the application bandwidth requirement threshold for using MPLS. Policies are a way of addressing cross cutting concerns and injecting more intelligence into the system. They can be applied at different levels of abstraction (types, instances, blueprints, everything). Policies themselves are a form of abstraction, albeit at a higher level than individual type abstractions.
The journey to higher levels of automation leads through our innate ability to use abstractions to simplify the real world (or virtual world). The drive to higher levels of abstraction in software led to object oriented programming, which ties state to related operations. Declarative programming takes the next step by getting closer to requirements driven systems, where implementation is completely hidden. Cloudify "DSL" is a general purpose language that enables the creation of declarative vocabularies, and defines an orchestrator role to operate on those vocabularies. From the base types, domain specific declarative vocabularies can be defined, which in turn greatly simplify the task of creating orchestrations. Going forward, declarative vocabularies will form part the building blocks for machine reasoning about intelligent orchestration.
1. Inversion Of Control Pattern↵
2. OASIS TOSCA↵
3. TOSCA NFV Profile↵