Autonomous K8s - Chapter 0: Architecture and Goal

Summary

  • Chapter 0: Architecture and Goal

Introduction

When I set out to build a self-managed Kubernetes cluster, I knew it had to be both resilient and adaptable. This chapter details the architecture that serves as the foundation of my setup, focusing on the infrastructure choices and the design principles that guided them. From selecting the physical servers to deciding on the right operating system for Kubernetes, each decision was made with the goal of achieving a robust, autonomous environment.

Choosing the Infrastructure

In an ideal scenario, I would have the resources to run my entire Kubernetes infrastructure on-premise, giving me full control over every aspect of the setup. However, the reality is that maintaining a completely on-premise solution requires significant investment in hardware, networking, and ongoing maintenance. While this remains my ultimate goal, I’ve opted for a more feasible solution by utilizing dedicated servers from OVH, allowing me to maintain a high level of control while avoiding the prohibitive costs of a full on-premise setup.

One of the key reasons I chose to run my Kubernetes cluster on self-managed dedicated servers rather than using a cloud provider is the desire for greater control and authority over my data. In today’s cloud-centric world, many companies are increasingly dependent on large tech providers to manage their critical infrastructure. While cloud services offer convenience and scalability, they often come at the cost of losing control over the data and the underlying infrastructure. This growing trend has raised concerns about privacy, data sovereignty, and the increasing influence of big tech companies.

By choosing dedicated servers, I retain full control over my environment, free from the constraints and potential vulnerabilities introduced by third-party cloud providers. OVH’s reliability and robust network infrastructure made it an ideal choice for hosting critical components of my cluster. The dedicated servers provide the raw horsepower and isolated environment necessary for running a production-grade Kubernetes setup without the overhead and limitations of shared resources.

The servers are interconnected with a private network, ensuring low-latency communication between nodes. This setup not only improves performance but also adds an extra layer of security by segregating cluster traffic from the public internet.

Virtualization with Proxmox

To make the most of each dedicated server, I opted to use Proxmox for virtualization. Proxmox is a powerful, open-source hypervisor that allows me to efficiently manage virtual machines and containers. On each server, I’ve created three VMs, each with a specific role:

  • Load Balancer VM: Handles traffic distribution across the cluster, ensuring high availability and efficient resource utilization.
  • Control Plane VM: Acts as the brain of the Kubernetes cluster, managing the state and operations of all other nodes.
  • Worker Node VM: Executes workloads, running the applications and services deployed in the cluster.

This virtualization strategy allows me to compartmentalize the different roles within the cluster, providing flexibility in management and scalability. Additionally, Proxmox’s ease of use and comprehensive feature set make it an ideal choice for managing the underlying infrastructure.

Talos as the Kubernetes OS

For the operating system, I selected Talos, a minimalist OS specifically designed for running Kubernetes. Talos strips away unnecessary components, reducing the attack surface and potential points of failure. Its immutable nature ensures that the operating system remains consistent and secure, with all configurations managed declaratively.

Choosing Talos aligns perfectly with my goal of creating an autonomous Kubernetes environment. It simplifies the operational aspects of managing the OS, allowing me to focus more on the cluster itself rather than the underlying infrastructure. Talos’s security-first approach and minimalistic design make it an ideal choice for running Kubernetes clusters in production.

Goal Architecture

To optimize connectivity and ensure high availability within my Kubernetes setup, I’m utilizing OVH’s vRack technology and IP Failover capabilities. The vRack provides a dedicated network interface for a private, isolated network between my servers, allowing secure, low-latency and high-speed communication without exposing internal traffic to the public internet.

In addition to the vRack, I’m leveraging OVH’s IP Failover feature, which operates independently of the vRack. IP Failover allows a public IP address to be reassigned to any of my servers, enabling seamless rerouting of traffic in the event of a server failure. This ensures that services remain accessible even during hardware issues, with the failover process being managed through a simple REST API call. This capability is critical for maintaining the availability and reliability of the public-facing components in my Kubernetes cluster.

In the Networking VM setup, I'll be utilizing Keepalived, HAProxy, and Nginx to ensure high availability and efficient load balancing for web services. The Networking VM will be configured to work in an active/passive failover mode. Keepalived will manage the failover mechanism, ensuring that if the active VM fails, the passive one seamlessly takes over. HAProxy will serve as the core load balancer, distributing incoming traffic across the backend servers to optimize resource use and prevent overloading any single server. Nginx will function as the web server, efficiently handling HTTP requests while providing reverse proxy services, caching, and additional load balancing at the application layer. This configuration ensures a robust, scalable, and resilient networking environment with minimal downtime.

Conclusion

This architecture forms the backbone of my autonomous Kubernetes cluster, balancing performance, security, and manageability. By leveraging dedicated servers, Proxmox, and Talos, I’ve laid the groundwork for a robust and self-sufficient environment. In the next chapter, I’ll dive deeper into the process of setting up Talos on these servers, detailing the steps and considerations that bring this architecture to life.