The Kubernetes Aware Admin (KAA) is a new feature released in NuoDB 4.1 and NuoDB Helm Charts 2.4.0. What is it?
It is reasonably easy to manage web apps, mobile backends, and trivial applications in Kubernetes. These are examples of stateless applications that can be managed using Kubernetes basic features such as Deployment controllers and usually do not require complex application-specific knowledge. Not long after Helm was introduced to the Kubernetes community, we learned that not every use case can be solved by deploying YAML templates. Stateful applications, and databases especially, are complex systems that require custom logic for operations such as back up, rolling upgrade, and failure recovery. To address the inherent complexity of managing stateful applications, Kubernetes Operators were introduced in 2016.
The functionality in this article is complementary to Operators.
A bit of NuoDB background
As a distributed SQL database, NuoDB is uniquely positioned within the Kubernetes space since NuoDB already has an administrative tier. It’s called NuoDB Admin, and as its name implies, it administers the database. Even before Kubernetes, NuoDB Admin managed the complexity of rolling upgrades, backup-and-restore, restart, and failure events while it protects the database from data loss. The Admin has been deployed across a variety of bare metal environments and has an extensive API to cover all operational needs. A Site Reliability Engineer (SRE) rarely interacts with the database processes (sometimes referred to as engines) directly. Instead, they can use the nuocmd command tool to issue domain modifications to NuoDB Admin.
Admin in Kubernetes
In a NuoDB deployment, the admin tier is deployed using a separate Helm chart that has to be installed before any database charts are installed. It is multi-tenant and manages all NuoDB databases in your cluster. Additionally, it can span multiple clusters in a multi-cluster multi-cloud environment while reducing the complexity of such deployments.
Kubernetes Aware Admin
“The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.” https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
I think of operators as domain aware scripts written in a variety of languages (GoLang, Java, Python). These scripts are external to the application and interact with the application by observing its state, or by calling into a well-known API. The KAA synchronizes the NuoDB control plane with the Kubernetes control plane in a way that is transparent to the user without requiring additional Kubernetes objects. The term Kubernetes operator usually refers to a Kubernetes extension that is external to the application and runs “on top of” the application. To avoid confusion by calling the new module the “Java Operator” we decided to pick the term Kubernetes Aware Admin. The KAA can simplify Kubernetes operations regardless of the method that was used to deploy NuoDB in your cluster. It can work with Operators, Helm 2, and Helm 3.
In Figure 1 you can see the difference between these two designs. The diagram shows the relevant actors and their communication paths to reach the desired state.
Reconciliation Loop
The basic premise of all Kubernetes Operators is that they execute a control loop to reconcile the desired state with the actual state. This is the reconciliation loop. In the case of NuoDB running in Kubernetes, there are multiple states that need to be reconciled:
- The Kubernetes desired state
- The Admin expected state
- The actual state of the engines
The KAA module reconciles all three states by inspecting the desired Kubernetes state and executing domain-specific commands to match the other two states.
The reconciliation loop has two phases. First, the Kubernetes Aware Admin collects state from all actors. Once an accurate image of the state has been established, the Admin will reconcile the current NuoDB state with the Kubernetes state and when necessary execute commands that will synchronize the NuoDB database state with Kubernetes, as seen in Figure 2.
Requirements
Since KAA is offering functionality that is similar to a Java Operator, it requires a similar set of permissions. The KAA never changes the state of Kubernetes and only ever reads Kubernetes state. All reconciliation is done one way. When installing the Admin Helm Chart, Helm will install a few additional Roles and RoleBindings. The name of the roles and bindings is nuodb-kube-inspector, to indicate the read-only nature of these requirements. The Kubernetes resources that are being watched are: StatefulSets, Deployments, PersistentVolumeClaims, and Pods. This list might be subject to change in future NuoDB releases.
Summary
The Kubernetes Aware Admin (KAA) is an operator-like module that enables tighter NuoDB integration in Kubernetes environments that improves reliability, ease of management, and overall application availability. It simplifies operation by reconciling the desired state in Kubernetes and executing NuoDB domain-specific commands to transition your databases into the correct and desired state.
Stay tuned for additional articles on the various operations that KAA executes in order to deliver a seamless always-on Kubernetes runtime environment for enterprise distributed SQL applications.
Adrian is the lead architect for the NuoDB Orchestration and Administration (ORCA) team, which develops tools for automatically managing NuoDB databases in various types of environments, from Kubernetes in the Cloud to bare-metal and on-prem. Adrian is a distributed systems expert with over a decade of experience building strongly-consistent, fault-tolerant services for managing and securing critical system infrastructure.