Building a Centralized Multi-cluster K8s ServiceBoard using K8s CRDs and Rancher
How and Why to Build a Centralized Multi-cluster K8s ServiceBoard
Background and Problem
We have hundreds of microservices running on more than 80 Kubernetes clusters on GCP (Google Cloud Platform) and Alibaba Cloud.
Getting K8s service status, version, Ingress URLs, and so on from multiple clusters is time-consuming and inefficient.
Solution
We want to build a Centralized Multi-cluster K8s ServiceBoard to help engineers quickly get K8s service status and information across all of our K8s clusters, such as service version, health status, K8s service name, Ingress URL, etc.
Here are 3 architectures for building it.
Architecture 1: Based on collectors in each K8s cluster
We could develop a collector and deploy it to every K8s cluster. A collector is a K8s customer controller that watches Services and Pods changes and collects k8s service information.
The architecture diagram is as follows:
However, we have more than 80 K8s clusters. It is very difficult to maintain so many collectors.
Architecture 2: Based on the Rancher platform
At Airwallex, we use Rancher as our unified K8s management platform. Rancher is a container management platform for managing multiple Kubernetes clusters. Rancher can communicate with all downstream K8s clusters via the tunnels established by the Rancher agents and server.
Therefore, we can connect all our K8s clusters with one collector using the Rancher platform.
The architecture diagram is as follows:
Maintaining 1 collector is far easier than maintaining 80 collectors. Based on Rancher, the service board server is more extensible, it can modify our k8s in the future.
Architecture 3: Based on Rancher + Kubernetes CRDs
Developing and maintaining Rancher Dashboard UI is not easy. However, Rancher Dashboard UI can display Kubernetes Custom Resource Definitions(CRDs).
Therefore, we can build a collector to watch changes in all clusters via Rancher proxy and store service information in Kubernetes Custom Resources. Then, the service information will be displayed in Rancher’s Dashboard UI.
The architecture diagram is as follows:
Result and Conclusion
Finally, we chose “Architecture 3: Based on Rancher + Kubernetes CRDs”. With Rancher Dashboard UI and Kubernetes CRDs, we no longer need to build frontend and backend services. Kubernetes and etcd serve as our backend server and database.
The Centralized K8s ServiceBoard pages are as follows:
ServiceBoard Overview
We can select multiple K8s clusters, search for services, and sort services by status on the ServiceBoard overview page.
ServiceBoard Detail Page
Click a service name, we can see the following page.
The spec field contains serviceInfomation , such as clusterName , service.name , service.namespace , service.ports, ingress.hosts, ingress.ip, pods.imageTags, pods.status and updatedAt.
The status field contains the service status.
All in all, with the introduction of the Centralized Multi-cluster K8s ServiceBoard, engineers can quickly and easily get all K8s service status and information from all of our K8s clusters.
