If you had to pick one property of a program that makes it easier to maintain, what would it be? Good documentation is certainly a candidate, but if the code itself is bad enough, that probably won’t help you. How about encapsulation? It can be liberating to know that the changes you’re making to a piece of code will only have local effects. However, what if there are other versions of that same code encapsulated elsewhere? Oops. To my mind, the most important property is lack of redundancy. The more places in the code that perform the same function or embody the same implied understanding, the harder it is to maintain without breaking something. This has been well-understood in the database world for decades; normalization is nearly a religious practice in relational database circles. The principle is important enough that it has a well-known acronym: DRY (Don’t Repeat Yourself). How might we apply this principle to Kubernetes manifests?
First of all, is this even a problem for Kubernetes manifests? For a single application, maybe not. The services, deployments, and other Kubernetes constructs will likely all be unique since any common behavior will be supported by replicated pods at runtime, not by replicated controllers in manifests. However… the ease of container-based deployment has made it common practice to run an application in a variety of environments. At Yipee.io, we run our own application in production, staging, and development environments, from public clouds to developer laptops. This ability to easily replicate a significant application is extremely powerful but brings with it the problem of maintaining several closely related but non-identical application configurations.
There have already been several attempts to address this problem within the industry. Here are some of the most prominent:
docker-composeprovides the option to use multiple configuration files (for Swarm applications). Each additional configuration file overrides any previous ones for singular values and merges sets of values with those from earlier files.
Helmsupports manifest templates that can be used to parameterize deployments. Sections of a YAML manifest can be marked as variables which are instantiated via separately specified values.
Ksonnetsupports programmatic configuration of Kubernetes apps via code in the extended JSON dialect Jsonnet.
Kustomizehas overlays that can be applied to base YAML manifests to produce specific configurations.
All of these work (with varying levels of convenience and power) but have issues:
docker-compose’s multiple configuration files only allow certain values to be changed and only in specific ways.
Helm templates require the base configuration to anticipate what parts might be changed by later overrides so that template variables can be inserted. Also, they require the user to learn an additional templating language in order to construct shared configurations.
Ksonnet uses a unique configuration language based on JSON. This forces users to learn a new formalism and makes configurations opaque for new users.
Kustomize’s overlays are just YAML which means users don’t have to learn a new language. However, the division of a model into many separate related files makes it hard to figure out just what a model contains and the overlays themselves cannot be read in isolation since they are updates to existing files.
The Yipee Approach
Yipee.io started out as a SaaS-based graphical modeling tool for building Kubernetes manifests and Docker Compose files. It was clear from the start that YAML, though capable of expressing complex configurations, left something to be desired as the sole representation of an application. In particular, though it’s relatively easy to modify an existing configuration for someone who knows how the application works, it’s difficult to see the big picture when looking at a large configuration in YAML. Yipee.io models, which represent Kubernetes objects graphically, make it far easier to understand the structure of an application. We believe the same holds true for managing multiple related configurations. We’re currently building an on-premise version of Yipee.io that runs in a user’s own Kubernetes cluster. Models in the on-premise Yipee.io can be derived from “parent” models and inherit their configurations. The derived models can then be modified but maintain their connections to their parents so that parent-level changes can be automatically propagated. The advantages of this approach include:
The model you’re working on is always complete. You don’t have to piece it together from chunks spread around the filesystem.
Graphical diffs between models (and versions of a single model) make it easy to see how a configuration has changed, even if large-scale changes have been made.
Upstream changes can be accepted/rejected in bulk or they can be addressed on a per-change basis. The system remembers when you change or remove a parent model object and won’t ask you to reprocess a change if you compare the same two models again.
Changes to ancestor models don’t accidentally break descendant models. Unless a user explicitly accepts upstream changes, the descendant models remain unchanged. If a user does want the changes, though, it’s trivial to accept them.
Subsystems can be included or embedded. If the subsystem is best viewed as a black box, it can be treated as if it were itself a Kubernetes object. If it’s more naturally seen as a set of top-level parts, it can be unpacked directly into the descendant model.
Contrast two representations of the same application. First, the Kubernetes manifests:
# Generated 2018-08-15T21:33:09.964Z by Yipee.io # Application: Parse-MongoDB # Last Modified: 2018-08-15T21:33:09.964Z apiVersion: v1 kind: Service metadata: name: parse-server annotations: yipee.io.modelId: 7adf4c72-4ca6-11e8-b87b-87cec0af93b2 yipee.io.modelURL: https://staging2.yipee.io/editor/7adf4c72-4ca6-11e8-b87b-87cec0af93b2/fc55cc2c-4251-11e8-b067-4f2de2893da6 yipee.io.contextId: fc55cc2c-4251-11e8-b067-4f2de2893da6 yipee.io.lastModelUpdate: '2018-04-30T18:46:32.106Z' spec: selector: run: parse-server ports: - port: 1337 targetPort: 1337 name: parse-server-1337 protocol: TCP nodePort: 31337 type: NodePort --- apiVersion: v1 kind: Service metadata: name: mongo annotations: yipee.io.modelId: 7adf4c72-4ca6-11e8-b87b-87cec0af93b2 yipee.io.modelURL: https://staging2.yipee.io/editor/7adf4c72-4ca6-11e8-b87b-87cec0af93b2/fc55cc2c-4251-11e8-b067-4f2de2893da6 yipee.io.contextId: fc55cc2c-4251-11e8-b067-4f2de2893da6 yipee.io.lastModelUpdate: '2018-04-30T18:46:32.106Z' spec: selector: app: mongo ports: - port: 27017 targetPort: 27017 name: peer protocol: TCP type: ClusterIP --- apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: mongo annotations: yipee.io.lastModelUpdate: '2018-08-15T21:33:09.920Z' yipee.io.modelId: edb88f9e-641e-11e8-8a4d-f3d7597e94c6 yipee.io.contextId: 89a94702-1558-11e7-a147-df26953bf272 yipee.io.modelURL: https://app.yipee.io/editor/edb88f9e-641e-11e8-8a4d-f3d7597e94c6/89a94702-1558-11e7-a147-df26953bf272 spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 0 selector: matchLabels: name: MongoDB component: mongo app: mongo template: spec: imagePullSecrets:  containers: - name: init-mongo image: mongo:3.4.1 command: - bash - /config/init.sh volumeMounts: - name: config mountPath: /config - name: mongodb ports: - containerPort: 27017 protocol: TCP name: web image: mongo:3.4.1 command: - mongod - --replSet - rs0 volumes: - name: config configMap: name: mongo-init terminationGracePeriodSeconds: 10 metadata: labels: name: MongoDB component: mongo app: mongo podManagementPolicy: OrderedReady replicas: 3 serviceName: mongo --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: parse-server annotations: yipee.io.lastModelUpdate: '2018-08-15T21:33:09.920Z' yipee.io.modelId: edb88f9e-641e-11e8-8a4d-f3d7597e94c6 yipee.io.contextId: 89a94702-1558-11e7-a147-df26953bf272 yipee.io.modelURL: https://app.yipee.io/editor/edb88f9e-641e-11e8-8a4d-f3d7597e94c6/89a94702-1558-11e7-a147-df26953bf272 spec: selector: matchLabels: name: MongoDB component: parse-server run: parse-server rollbackTo: revision: 0 template: spec: containers: - name: parse-server ports: - containerPort: 1337 protocol: TCP name: parse-server image: parseplatform/parse-server env: - name: PARSE_SERVER_APPLICATION_ID value: my-app-id - name: PARSE_SERVER_DATABASE_URI value: mongodb://mongo-0.mongo:27017,mongo-1.mongo:27017,mongo-2.mongo:27017/dev?replicaSet=rs0 - name: PARSE_SERVER_MASTER_KEY value: my-master-key restartPolicy: Always imagePullSecrets:  metadata: labels: name: MongoDB component: parse-server run: parse-server strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 1 replicas: 1 revisionHistoryLimit: 2
Here’s the Yipee.io version:
I think it’s clear that the graphical view helps make the big picture evident. You can see instantly that the application is built from three container types, one deployment, one stateful set, and two services. You can also see how they’re related. If the structure of the application in an ancestor model changes, a graphical diff will make it obvious what changed and what depends on the change.
The rise of container-based deployment has led many companies to construct multiple variants of their application configurations. These variants have significant commonality and generate the same issues around redundancy that software developers and database administrators have dealt with for decades. The issues are serious enough that, even though Kubernetes is a relatively young technology, multiple players in the industry have attempted to address them with varying levels of success. Yipee.io is developing a graphical approach to managing Kubernetes models that supports embedded models, model inheritance, and model versioning with an intuitive visual editor and graphical diff capability. The graphical diff makes it easy to manage derived models and propagate changes from ancestors to descendants.
If anything you’ve read here sounds interesting to you, please visit our DRY Kubernetes page and fill out the form to let us know. We’re looking for early adopters to help ensure that we’re solving everyone’s problems and not just our own. Also, if you’re a fan of GraphQL, you might want to check out our Kubernetes GraphQL interface that we’re building in support of our on-premise product: Kubeiql.
Thanks for reading!