Subsections of Operator
KubeBuilder
Basic
Kubebuilder 是一个使用 CRDs 构建 K8s API 的 SDK,主要是:
- 基于 controller-runtime 以及 client-go 构建
- 提供一套可扩展的 API 框架,方便用户从零开始开发 CRDs 和 Controllers 和 Admission Webhooks 来扩展 K8s。
- 还提供脚手架工具初始化 CRDs 工程,自动生成 boilerplate 模板代码和配置;
Architecture
Main.go
import (
_ "k8s.io/client-go/plugin/pkg/client/auth"
ctrl "sigs.k8s.io/controller-runtime"
)
// nolint:gocyclo
func main() {
...
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{}
...
if err = (&controller.GuestbookReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "Guestbook")
os.Exit(1)
}
...
if os.Getenv("ENABLE_WEBHOOKS") != "false" {
if err = webhookwebappv1.SetupGuestbookWebhookWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create webhook", "webhook", "Guestbook")
os.Exit(1)
}
}
Manager
Manager是核心组件,可以协调多个控制器、处理缓存、客户端、领导选举等,来自https://github.com/kubernetes-sigs/controller-runtime/blob/v0.20.0/pkg/manager/manager.go
- Client 承担了与 Kubernetes API Server 通信、操作资源对象、读写缓存等关键职责; 分为两类:
- Reader:优先读Cache, 避免频繁访问 API Server, Get后放缓存
- Writer: 支持写操作(Create、Update、Delete、Patch),直接与 API Server 交互。
- informers 是 client-go 提供的核心组件,用于监听(Watch)Kubernetes API Server 中特定资源类型(如 Pod、Deployment 或自定义 CRD)的变更事件(Create/Update/Delete)。
- Client 依赖 Informer 机制自动同步缓存。当 API Server 中资源变更时,Informer 会定时更新本地缓存,确保后续读操作获取最新数据。
- Cache
- Cache 通过 内置的client 的 ListWatcher机制 监听 API Server 的资源变更。
- 事件被写入本地缓存(如 Indexer),避免频繁访问 API Server。
- 缓存(Cache)的作用是减少对API Server的直接请求,同时保证控制器能够快速读取资源的最新状态。
- Event
Kubernetes API Server 通过 HTTP 长连接 推送资源变更事件,client-go 的 Informer 负责监听这些消息。
- Event:事件是Kubernetes API Server与Controller之间传递的信息,包含资源类型、资源名称、事件类型(ADDED、MODIFIED、DELETED)等信息,并转换成requets, check link
- API Server → Manager的Informer → Cache → Controller的Watch → Predicate过滤 → WorkQueue → Controller的Reconcile()方法
Controller
It’s a controller’s job to ensure that, for any given object the actual state of the world matches the desired state in the object. Each controller focuses on one root Kind, but may interact with other Kinds.
func (r *GuestbookReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
...
}
func (r *GuestbookReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&webappv1.Guestbook{}).
Named("guestbook").
Complete(r)
}
If you wanna build your own controller, please check https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md
每个Controller在初始化时会向Manager注册它关心的资源类型(例如通过Owns(&v1.Pod{})声明关注Pod资源)。
Manager根据Controller的注册信息,为相关资源创建对应的Informer和Watch, check link
当资源变更事件发生时,Informer会将事件从缓存中取出,并通过Predicate(过滤器)判断是否需要触发协调逻辑。
若事件通过过滤,Controller会将事件加入队列(WorkQueue),最终调用用户实现的Reconcile()函数进行处理, check link
func (c *Controller[request]) Start(ctx context.Context) error {
c.ctx = ctx
queue := c.NewQueue(c.Name, c.RateLimiter)
c.Queue = &priorityQueueWrapper[request]{TypedRateLimitingInterface: queue}
err := func() error {
// start to sync event sources
if err := c.startEventSources(ctx); err != nil {
return err
}
for i := 0; i < c.MaxConcurrentReconciles; i++ {
go func() {
for c.processNextWorkItem(ctx) {
}
}()
}
}()
c.LogConstructor(nil).Info("All workers finished")
}
func (c *Controller[request]) processNextWorkItem(ctx context.Context) bool {
obj, priority, shutdown := c.Queue.GetWithPriority()
c.reconcileHandler(ctx, obj, priority)
}
Webhook
Webhooks are a mechanism to intercept requests to the Kubernetes API server. They can be used to validate, mutate, or even proxy requests.
func (d *GuestbookCustomDefaulter) Default(ctx context.Context, obj runtime.Object) error {}
func (v *GuestbookCustomValidator) ValidateCreate(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {}
func (v *GuestbookCustomValidator) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Object) (admission.Warnings, error) {}
func (v *GuestbookCustomValidator) ValidateDelete(ctx context.Context, obj runtime.Object) (admission.Warnings, error) {}
func SetupGuestbookWebhookWithManager(mgr ctrl.Manager) error {
return ctrl.NewWebhookManagedBy(mgr).For(&webappv1.Guestbook{}).
WithValidator(&GuestbookCustomValidator{}).
WithDefaulter(&GuestbookCustomDefaulter{}).
Complete()
}
Links
Subsections of KubeBuilder
Quick Start
Prerequisites
- go version v1.23.0+
- docker version 17.03+.
- kubectl version v1.11.3+.
- Access to a Kubernetes v1.11.3+ cluster.
Installation
# download kubebuilder and install locally.
curl -L -o kubebuilder "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder && sudo mv kubebuilder /usr/local/bin/
Create A Project
mkdir -p ~/projects/guestbook
cd ~/projects/guestbook
kubebuilder init --domain my.domain --repo my.domain/guestbook
Create An API
kubebuilder create api --group webapp --version v1 --kind Guestbook
Prepare a K8s Cluster
minikube start --kubernetes-version=v1.27.10 --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --cpus=4 --memory=4g --disk-size=50g --force
asdasda
Install CRDs
check installed crds in k8s
kubectl get crds
install guestbook
crd in k8s
cd ~/projects/guestbook
make install
uninstall CRDs
make uninstall
make undeploy
Deploy to cluster
make docker-build IMG=aaron666/guestbook-operator:test
make docker-build docker-push IMG=<some-registry>/<project-name>:tag
make deploy IMG=<some-registry>/<project-name>:tag