Tech on Alan Zhan Blog

Building a Private GKE Cluster with Terraform

Sun, 12 Mar 2023 18:04:41 +0800

Let’s spend about ten minutes together — I’ll walk you through using Terraform to build a private GKE Cluster step by step!

Setting Up GCP

Create a Service Account

First, go to the Google Cloud console and create a Service Account for Terraform to use. Follow steps 1, 2, and 3 in the diagram above. If you already have a Service Account, you can skip this step.

Kubernetes Core Component: Kubelet

Tue, 30 Aug 2022 20:56:08 +0800

Kubelet Architecture

As shown in the kubelet internal component structure diagram below, Kubelet is composed of many internal components:

Kubelet API: Including the authenticated API on port 10250, cAdvisor API on port 4194, read-only API on port 10255, and health check API on port 10248.
syncLoop: Receives Pod updates from the API or manifest directories, sends them to podWorkers for processing, extensively using channels for async request handling.
Auxiliary managers: Such as cAdvisor, PLEG, Volume Manager, etc., handling work outside of syncLoop.
CRI: Container Runtime Interface, responsible for communicating with container runtime shims.
Container runtimes: Such as dockershim, rkt, etc.
Network plugins: Currently supports CNI and kubenet.

Kubernetes Core Component: Controller Manager

Sat, 30 Jul 2022 20:07:01 +0800

Controller Manager is the automation control center of a Kubernetes cluster, containing over 30 controllers that manage Pod-related, network-related, storage-related operations, and more. Most controllers work similarly — each controller is a control loop responsible for watching its corresponding resources through the API server, deciding on the next action based on the object’s state, and driving it toward the desired state.

Controller Manager is the brain of the cluster and the key to keeping it running.
Its role is to ensure Kubernetes follows the declarative system specification, keeping the system’s Actual State consistent with the user-defined Desired State.
Controller Manager is a combination of multiple controllers. Each controller is a control loop responsible for watching its managed objects and completing configuration when objects change.
Failed controller configurations typically trigger automatic retries. Through the controller’s continuous retry mechanism, the entire cluster ensures Eventual Consistency.

Controller Workflow

Kubernetes - Node Maintenance and Pod Migration

Sun, 10 Jul 2022 18:19:11 +0800

Our company recently discovered machines on GKE with particularly low resource utilization. Those unused resources are still costing money, so we created a new Node Pool with lower and cheaper resources, then used Cordon + Drain (or manually deleted Pods) to reschedule Pods onto the new Node Pool. On GKE, simply deleting the old Node Pool would achieve the same result, but we chose the safer approach.

This essentially marks a Node as unschedulable, then lets Pods migrate to the expected hosts. Let’s discuss how to safely handle node failures or upgrades.

Kubernetes Core Component: Scheduler

Mon, 16 May 2022 22:40:48 +0800

Strictly speaking, the Scheduler is a special type of Controller — its working principle is no different from other controllers.

The Scheduler’s special responsibility is to monitor all unscheduled Pods in the cluster and obtain the health status and resource usage of all nodes, selecting the best node for each pending Pod to complete scheduling.

kube-scheduler is responsible for assigning and scheduling Pods to nodes within the cluster. It watches kube-apiserver for Pods that haven’t been assigned to a Node, then assigns nodes to these Pods based on scheduling policies (by updating the Pod’s NodeName field).

Kubernetes Core Component: API Server

Sun, 24 Apr 2022 19:07:12 +0800

API Server

kube-apiserver is one of the most important core components of Kubernetes, providing the following key features:

Cluster management REST API, including authentication, authorization, data validation, and cluster state changes
- Authentication
- Authorization
- Admission (Mutating & Validating)
Acts as the data exchange and communication hub between other modules. Other modules can only query or modify data through the API Server — only the API Server can directly operate on etcd.

kube-apiserver supports HTTPS (default port 6443) and HTTP API (default listening on 127.0.0.1:8080). The HTTP API is insecure with no authentication or authorization mechanism — it’s not recommended for production environments.

MongoDB Index Best Practices

Sun, 10 Apr 2022 16:19:10 +0800

After the sorted operation OOM incident in production from the previous post, I realized my understanding of MongoDB indexes wasn’t as deep as it should be. I started searching extensively and finally organized some key points. Some were covered in the previous post — feel free to review MongoDB Sorted Operation OOM.

When to Use Indexes

When you have sorting scenarios and in-memory sorting exceeds 32 MB.
When a specific field has a uniqueness requirement.
When the document count is large.
Build indexes on high-cardinality fields.

When NOT to Use Indexes

When an index cannot effectively filter data.
Don’t set indexes on frequently updated fields.

How to Design and Use Indexes

Use Compound Indexes Instead of Single Field Indexes

In MongoDB, a find query can only use one index at a time (in most scenarios). So if your use case frequently filters on multiple fields, use a Compound Index composed of multiple fields to match your query conditions.

MongoDB Sorted Operation OOM

Sun, 20 Mar 2022 15:11:15 +0800

This is a rare case where I learned by working backward from a problem — I’d better document it well. Recently, our production environment hit a MongoDB sort operation limit with this specific error:

MongoDB.Driver.MongoCommandException: Command find failed: Encountered non-retryable error during query :: caused by :: Executor error during find command :: caused by :: Sort operation used more than the maximum 33554432 bytes of RAM.

Well, the problem has occurred. Let’s solve it first — so we added an index as an immediate fix. But what exactly is a MongoDB index?

Kubernetes Core Component: etcd

Mon, 28 Feb 2022 15:58:47 +0800

In any system, what is the most important thing? The answer is simple: data. So I’m starting my deep dive with Kubernetes’ database — etcd!

What is etcd?

etcd is a distributed key-value store developed by CoreOS based on the Raft algorithm. It can be used for service discovery, shared configuration, and consistency guarantees (such as database leader election, distributed locks, etc.).

In distributed systems, managing state across nodes has always been challenging. etcd is designed specifically for service discovery and registration in cluster environments. It provides features like data TTL expiration, data change monitoring, multi-value operations, directory watching, distributed lock atomic operations, and more — making it easy to track and manage cluster node states.

Installing Kubernetes with Kubeadm - Twice

Sun, 20 Feb 2022 15:04:19 +0800

To better understand K8s through hands-on practice, I wanted to set up a K8s cluster myself. There are increasingly many ways to self-host K8s nowadays — minikube, Kubeadm, and more. I decided to go with Kubeadm, but the installation wasn’t exactly smooth. This post shares the problems I encountered and how I solved them.

First Installation of Kubeadm

Install Ubuntu.
Install Docker.
1. Modify Docker’s cgroup settings to ensure the K8s and container runtime cgroup drivers match, preventing system instability.
```
sudo vim /etc/docker/daemon.json
```
```
{
    "exec-opts": ["native.cgroupdriver=systemd"]
}
```
2. Restart Docker
```
sudo systemctl restart docker
```

Install Kubeadm.

Allow iptables to inspect bridged traffic.

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

Update apt and install packages required by K8s.

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

Download the Google Cloud public signing key.

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Add the K8s apt repository.

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update the apt package index, install kubelet, kubeadm, and kubectl, and pin their versions.

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Initialize K8s.
```
sudo kubeadm init
```

Join worker nodes

sudo kubeadm join 192.168.83.130:6443 --token token....  --discovery-token-ca-cert-hash sha256:......................

Copy kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

K8s Nodes Not Ready

I finally got it installed! So I added the worker nodes to the cluster. After joining, just as I was about to start using K8s, I noticed all nodes were in NotReady status.

Kubernetes Fundamentals

Mon, 14 Feb 2022 21:48:40 +0800

It’s finally time to dive deep into Kubernetes. Since Kubernetes is built with Go, I’ve already done some in-depth research on Go beforehand. I won’t be covering Docker and container technology here since I’ve already studied those extensively. If you’d like to see that content, let me know! From here on, I’ll be writing a series of articles focused on Kubernetes.

Kubernetes

What is Kubernetes (K8s)? K8s is the abbreviation for Kubernetes — because there are eight letters between “K” and “s.” It’s Google’s open-source container cluster management system, the open-source version of Google’s years of large-scale container management technology called Borg. Its main features include:

Golang Memory Management and GC In-Depth Analysis

Sun, 13 Feb 2022 14:42:09 +0800

A new year has arrived! After completing the analysis of Golang Goroutine and GMP Model In-Depth Analysis, I gained a much more comprehensive understanding of the Go language. But after learning about GMP, aren’t we still missing the memory management aspect? So today, let’s dive deep into how Go manages its memory.

The Memory Management Debate

When it comes to memory management, there’s been a long-standing debate: who should manage memory — the machine or the developer? Whether it’s machine-managed or human-managed, everyone agrees that memory management is critically important, but opinions diverge:

Golang Goroutine and GMP Model In-Depth Analysis

Mon, 24 Jan 2022 20:24:58 +0800

I’ve been studying Kubernetes recently, so I need to become good friends with the Go language. While reading, I came across goroutines, but couldn’t fully understand where they came from and why they exist. So before we dive deep into goroutines, we should first learn some history — this will give us a more comprehensive understanding of the principles and design philosophy behind goroutines.

The Origin of Golang’s Scheduler

The Single-Process Era

We know that software runs on top of the operating system, and the CPU does the actual computation. In early operating systems, each program was a process, and the next process could only run after the current one finished.

10 Dockerfile Best Practices

Sun, 16 Jan 2022 16:55:15 +0800

I was recently organizing Docker knowledge for myself and came across Dockerfile best practices. I planned to summarize them, but then I found that the official documentation already has best practices, so I decided to translate them and add my own insights.

Containers Should Be Ephemeral

Containers built from a Dockerfile should be as ephemeral as possible. “Ephemeral” here means they can be started quickly and stopped quickly.

Understand the Build Context

Including files that aren’t needed to build the image results in a larger build context and larger image. This increases build time, pull/push time, and the runtime size of the container.

GitHub Pages Custom Domain Setup

Sun, 09 Jan 2022 14:58:58 +0800

I recently switched my domain name, so I decided to take some notes to help everyone easily set up their own GitHub Pages custom domain.

Step 1: Purchase a Domain

First, purchase the domain name you want. I bought mine from Google Domains. If you have a preferred domain registrar, feel free to use that. This guide uses Google Domains as an example.

Step 2: Add a CNAME Record in DNS Settings

After purchasing the domain, you need to configure the DNS settings. Follow the diagram below to set things up.