☸️Kubernetes 的子部分
Prepare k8s Cluster
There are many ways to build a kubernetes cluster.
Install Kuberctl
MIRROR="files.m.daocloud.io/"
VERSION=$(curl -L -s https://${MIRROR}dl.k8s.io/release/stable.txt)
[ $(uname -m) = x86_64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/amd64/kubectl"
[ $(uname -m) = aarch64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/arm64/kubectl"
chmod u+x kubectl
mkdir -p ${HOME}/bin
mv -f kubectl ${HOME}/bin
Build Cluster
MIRROR="files.m.daocloud.io/"
VERSION=v0.20.0
[ $(uname -m) = x86_64 ] && curl -sSLo kind "https://${MIRROR}github.com/kubernetes-sigs/kind/releases/download/${VERSION}/kind-linux-amd64"
[ $(uname -m) = aarch64 ] && curl -sSLo kind "https://${MIRROR}github.com/kubernetes-sigs/kind/releases/download/${VERSION}/kind-linux-arm64"
chmod u+x kind
mkdir -p ${HOME}/bin
mv -f kind ${HOME}/bin
Creating a Kubernetes cluster is as simple as kind create cluster
kind create cluster --name test
and the you can visit https://kind.sigs.k8s.io/docs/user/quick-start/ for mode detail.
MIRROR="files.m.daocloud.io/"
[ $(uname -m) = x86_64 ] && curl -sSLo minikube "https://${MIRROR}storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64"
[ $(uname -m) = aarch64 ] && curl -sSLo minikube "https://${MIRROR}storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64"
chmod u+x minikube
mkdir -p ${HOME}/bin
mv -f minikube ${HOME}/bin
[Optional] disable aegis service and reboot system for aliyun
sudo systemctl disable aegis && sudo reboot
after you download binary, you can start your cluster
minikube start --kubernetes-version=v1.27.10 --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --cpus=6 --memory=24g --disk-size=100g
add alias for convinence
alias kubectl="minikube kubectl --"
and then you can visit https://minikube.sigs.k8s.io/docs/start/ for more detail.
Prerequisites
Hardware Requirements:
- At least 2 GB of RAM per machine (minimum 1 GB)
- 2 CPUs on the master node
- Full network connectivity among all machines (public or private network)
Operating System:
- Ubuntu 20.04/18.04, CentOS 7/8, or any other supported Linux distribution.
Network Requirements:
- Unique hostname, MAC address, and product_uuid for each node.
- Certain ports need to be open (e.g., 6443, 2379-2380, 10250, 10251, 10252, 10255, etc.)
Disable Swap:
sudo swapoff -a
Steps to Setup Kubernetes Cluster
- Prepare Your Servers Update the Package Index and Install Necessary Packages On all your nodes (both master and worker):
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl
Add the Kubernetes APT Repository
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
Install kubeadm, kubelet, and kubectl
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
- Initialize the Master Node On the master node, initialize the Kubernetes control plane:
sudo kubeadm init --pod-network-cidr=192.168.0.0/16
The –pod-network-cidr flag is used to set the Pod network range. You might need to adjust this based on your network provider
Set up Local kubeconfig
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- Install a Pod Network Add-on You can install a network add-on like Flannel, Calico, or Weave. For example, to install Calico:
- Join Worker Nodes to the Cluster On each worker node, run the kubeadm join command provided at the end of the kubeadm init output on the master node. It will look something like this:
sudo kubeadm join <master-ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>
If you lost the join command, you can create a new token on the master node:
sudo kubeadm token create --print-join-command
- Verify the Cluster Once all nodes have joined, you can verify the cluster status from the master node:
kubectl get nodes
This command should list all your nodes with the status “Ready”.
Command
Command 的子部分
Kubectl CheatSheet
Switch Context
- use different config
kubectl --kubeconfig /root/.kube/config_ack get pod
Resource
create resource
Resource Fromkubectl create -n <$namespace> -f <$file_url>
helm install <$resource_id> <$resource_id> \ --namespace <$namespace> \ --create-namespace \ --version <$version> \ --repo <$repo_url> \ --values resource.values.yaml \ --atomic
debug resource
kubectl -n <$namespace> describe <$resource_id>
- logging resource
kubectl -n <$namespace> logs -f <$resource_id>
- port forwarding resource
kubectl -n <$namespace> port-forward <$resource_id> --address 0.0.0.0 8080:80 # local:pod
- delete all resource under specific namespace
kubectl delete all --all -n <$namespace>
- delete error pods
kubectl -n <$namespace> delete pods --field-selector status.phase=Failed
- force delete
kubectl -n <$namespace> delete pod <$resource_id> --force --grace-period=0
- opening a Bash Shell inside a Pod
kubectl -n <$namespace> exec -it <$resource_id> -- bash
- copy secret to another namespace
kubectl -n <$namespaceA> get secret <$secret_name> -o json \
| jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
| kubectl -n <$namespaceB> apply -f -
- copy secret to another name
kubectl -n <$namespace> get secret <$old_secret_name> -o json | \
jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid","ownerReferences","annotations","labels"]) | .metadata.name = "<$new_secret_name>"' | \
kubectl apply -n <$namespace> -f -
- delete all completed job
kubectl delete jobs -n <$namespace> --field-selector status.successful=1
Nodes
- add taint
kubectl taint nodes <$node_ip> <key:value>
- remove taint
kubectl remove taint
- show info extract by json path
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
Deploy
- rollout show rollout history
kubectl -n <$namespace> rollout history deploy/<$deploy_resource_id>
undo rollout
kubectl -n <$namespace> rollout undo deploy <$deploy_resource_id> --to-revision=1
Container
Container 的子部分
CheatShett
- remove specific image
podman rmi <$image_id>
- remove all
<none>
images
podman rmi `podamn images | grep '<none>' | awk '{print $3}'`
- remove all stopped containers
podman container prune
- remove all docker images not used
podman image prune
- find ip address of a container
podman inspect --format='{{.NetworkSettings.IPAddress}}' minio-server
- exec into container
podman run -it <$container_id> /bin/bash
- run with environment
podman run -d --replace
-p 18123:8123 -p 19000:9000 \
--name clickhouse-server \
-e ALLOW_EMPTY_PASSWORD=yes \
--ulimit nofile=262144:262144 \
quay.m.daocloud.io/kryptonite/clickhouse-docker-rootless:20.9.3.45
--ulimit nofile=262144:262144
: 262144 is the maximum users process or for showing maximum user process limit for the logged-in user
ulimit
is admin access required Linux shell command which is used to see, set, or limit the resource usage of the current user. It is used to return the number of open file descriptors for each process. It is also used to set restrictions on the resources used by a process.
- login registry
podman login --tls-verify=false --username=ascm-org-1710208820455 cr.registry.res.cloud.zhejianglab.com -p 'xxxx'
- tag image
podman tag 76fdac66291c cr.registry.res.cloud.zhejianglab.com/ay-dev/datahub-s3-fits:1.0.0
- push image
podman push cr.registry.res.cloud.zhejianglab.com/ay-dev/datahub-s3-fits:1.0.0
- remove specific image
docker rmi <$image_id>
- remove all
<none>
images
docker rmi `docker images | grep '<none>' | awk '{print $3}'`
- remove all stopped containers
docker container prune
- remove all docker images not used
docker image prune
- find ip address of a container
docker inspect --format='{{.NetworkSettings.IPAddress}}' minio-server
- exec into container
docker exec -it <$container_id> /bin/bash
- run with environment
docker run -d --replace -p 18123:8123 -p 19000:9000 --name clickhouse-server -e ALLOW_EMPTY_PASSWORD=yes --ulimit nofile=262144:262144 quay.m.daocloud.io/kryptonite/clickhouse-docker-rootless:20.9.3.45
--ulimit nofile=262144:262144
: sssss
copy file
Copy a local file into container
docker cp ./some_file CONTAINER:/work
or copy files from container to local path
docker cp CONTAINER:/var/logs/ /tmp/app_logs
load a volume
docker run --rm \
--entrypoint bash \
-v $PWD/data:/app:ro \
-it docker.io/minio/mc:latest \
-c "mc --insecure alias set minio https://oss-cn-hangzhou-zjy-d01-a.ops.cloud.zhejianglab.com/ g83B2sji1CbAfjQO 2h8NisFRELiwOn41iXc6sgufED1n1A \
&& mc --insecure ls minio/csst-prod/ \
&& mc --insecure mb --ignore-existing minio/csst-prod/crp-test \
&& mc --insecure cp /app/modify.pdf minio/csst-prod/crp-test/ \
&& mc --insecure ls --recursive minio/csst-prod/"
Template
Template 的子部分
DevContainer Template
DEV
DEV 的子部分
Devpod
Preliminary
Get provider config
# just copy ~/.kube/config
for example, the original config
apiVersion: v1
clusters:
- cluster:
certificate-authority: <$file_path>
extensions:
- extension:
provider: minikube.sigs.k8s.io
version: v1.33.0
name: cluster_info
server: https://<$minikube_ip>:8443
name: minikube
contexts:
- context:
cluster: minikube
extensions:
- extension:
provider: minikube.sigs.k8s.io
version: v1.33.0
name: context_info
namespace: default
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
client-certificate: <$file_path>
client-key: <$file_path>
you need to rename clusters.cluster.certificate-authority
, clusters.cluster.server
, users.user.client-certificate
, users.user.client-key
.
clusters.cluster.certificate-authority -> clusters.cluster.certificate-authority-data
clusters.cluster.server -> clusters.cluster.server-data
users.user.client-certificate -> users.user.client-certificate-data
users.user.client-key -> users.user.client-key-data
the data you paste after each key should be base64
cat <$file_path> | base64
then we should forward minikube port in your own pc
#where you host minikube
MACHINE_IP_ADDRESS=10.200.60.102
USER=ayay
MINIKUBE_IP_ADDRESS=$(ssh -o 'UserKnownHostsFile /dev/null' $USER@$MACHINE_IP_ADDRESS '$HOME/bin/minikube ip')
ssh -o 'UserKnownHostsFile /dev/null' $USER@$MACHINE_IP_ADDRESS -L "*:8443:$MINIKUBE_IP_ADDRESS:8443" -N -f
then, modified config file should be look like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxxxxxxxxxxx
extensions:
- extension:
provider: minikube.sigs.k8s.io
version: v1.33.0
name: cluster_info
server: https://127.0.0.1:8443
name: minikube
contexts:
- context:
cluster: minikube
extensions:
- extension:
provider: minikube.sigs.k8s.io
version: v1.33.0
name: context_info
namespace: default
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
client-certificate: xxxxxxxxxxxx
client-key: xxxxxxxxxxxxxxxx
Create workspace
- get git repo link
- choose appropriate provider
- choose ide type and version
- and go!
Useful Command
download kubectl binary
MIRROR="files.m.daocloud.io/"
VERSION=$(curl -L -s https://${MIRROR}dl.k8s.io/release/stable.txt)
[ $(uname -m) = x86_64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/amd64/kubectl"
[ $(uname -m) = aarch64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/arm64/kubectl"
chmod u+x kubectl
mkdir -p ${HOME}/bin
mv -f kubectl ${HOME}/bin
Everything works fine.
when you in pod, and using kubectl you should change clusters.cluster.server
in ~/.kube/config
to https://<$minikube_ip>:8443
download argocd binary
MIRROR="files.m.daocloud.io/"
VERSION=v2.9.3
[ $(uname -m) = x86_64 ] && curl -sSLo argocd "https://${MIRROR}github.com/argoproj/argo-cd/releases/download/${VERSION}/argocd-linux-amd64"
[ $(uname -m) = aarch64 ] && curl -sSLo argocd "https://${MIRROR}github.com/argoproj/argo-cd/releases/download/${VERSION}/argocd-linux-arm64"
chmod u+x argocd
mkdir -p ${HOME}/bin
mv -f argocd ${HOME}/bin
exec into devpod
kubectl -n devpod exec -it <$resource_id> -c devpod -- bin/bash
add DNS item
10.102.1.52 gitee.zhejianglab.com
Deploy
Operator
Operator 的子部分
KubeBuilder
Proxy
Proxy 的子部分
Daocloud
1. install container tools
systemctl stop firewalld && systemctl disable firewalld
sudo dnf install -y podman
podman run -d -P m.daocloud.io/docker.io/library/nginx