☸️Kubernetes 的子部分

Prepare k8s Cluster

    There are many ways to build a kubernetes cluster.

    Install Kuberctl

    MIRROR="files.m.daocloud.io/"
    VERSION=$(curl -L -s https://${MIRROR}dl.k8s.io/release/stable.txt)
    [ $(uname -m) = x86_64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/amd64/kubectl"
    [ $(uname -m) = aarch64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/arm64/kubectl"
    chmod u+x kubectl
    mkdir -p ${HOME}/bin
    mv -f kubectl ${HOME}/bin

    Build Cluster

    MIRROR="files.m.daocloud.io/"
    VERSION=v0.20.0
    [ $(uname -m) = x86_64 ] && curl -sSLo kind "https://${MIRROR}github.com/kubernetes-sigs/kind/releases/download/${VERSION}/kind-linux-amd64"
    [ $(uname -m) = aarch64 ] && curl -sSLo kind "https://${MIRROR}github.com/kubernetes-sigs/kind/releases/download/${VERSION}/kind-linux-arm64"
    chmod u+x kind
    mkdir -p ${HOME}/bin
    mv -f kind ${HOME}/bin

    Creating a Kubernetes cluster is as simple as kind create cluster

    kind create cluster --name test

    and the you can visit https://kind.sigs.k8s.io/docs/user/quick-start/ for mode detail.

    MIRROR="files.m.daocloud.io/"
    [ $(uname -m) = x86_64 ] && curl -sSLo minikube "https://${MIRROR}storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64"
    [ $(uname -m) = aarch64 ] && curl -sSLo minikube "https://${MIRROR}storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64"
    chmod u+x minikube
    mkdir -p ${HOME}/bin
    mv -f minikube ${HOME}/bin

    [Optional] disable aegis service and reboot system for aliyun

    sudo systemctl disable aegis && sudo reboot

    after you download binary, you can start your cluster

    minikube start --kubernetes-version=v1.27.10 --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --cpus=6 --memory=24g --disk-size=100g

    add alias for convinence

    alias kubectl="minikube kubectl --"

    and then you can visit https://minikube.sigs.k8s.io/docs/start/ for more detail.

    Prerequisites

    • Hardware Requirements:

      1. At least 2 GB of RAM per machine (minimum 1 GB)
      2. 2 CPUs on the master node
      3. Full network connectivity among all machines (public or private network)
    • Operating System:

      1. Ubuntu 20.04/18.04, CentOS 7/8, or any other supported Linux distribution.
    • Network Requirements:

      1. Unique hostname, MAC address, and product_uuid for each node.
      2. Certain ports need to be open (e.g., 6443, 2379-2380, 10250, 10251, 10252, 10255, etc.)
    • Disable Swap:

      sudo swapoff -a

    Steps to Setup Kubernetes Cluster

    1. Prepare Your Servers Update the Package Index and Install Necessary Packages On all your nodes (both master and worker):
    sudo apt-get update
    sudo apt-get install -y apt-transport-https ca-certificates curl

    Add the Kubernetes APT Repository

    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
    deb http://apt.kubernetes.io/ kubernetes-xenial main
    EOF

    Install kubeadm, kubelet, and kubectl

    sudo apt-get update
    sudo apt-get install -y kubelet kubeadm kubectl
    sudo apt-mark hold kubelet kubeadm kubectl
    1. Initialize the Master Node On the master node, initialize the Kubernetes control plane:
    sudo kubeadm init --pod-network-cidr=192.168.0.0/16

    The –pod-network-cidr flag is used to set the Pod network range. You might need to adjust this based on your network provider

    Set up Local kubeconfig

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    1. Install a Pod Network Add-on You can install a network add-on like Flannel, Calico, or Weave. For example, to install Calico:
    1. Join Worker Nodes to the Cluster On each worker node, run the kubeadm join command provided at the end of the kubeadm init output on the master node. It will look something like this:
    sudo kubeadm join <master-ip>:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

    If you lost the join command, you can create a new token on the master node:

    sudo kubeadm token create --print-join-command
    1. Verify the Cluster Once all nodes have joined, you can verify the cluster status from the master node:
    kubectl get nodes

    This command should list all your nodes with the status “Ready”.

    Command 的子部分

    Kubectl CheatSheet

    Switch Context

    • use different config
    kubectl --kubeconfig /root/.kube/config_ack get pod

    Resource

    • create resource

      Resource From
        kubectl create -n <$namespace> -f <$file_url>
      apiVersion: v1
      kind: Service
      metadata:
      labels:
          app.kubernetes.io/component: server
          app.kubernetes.io/instance: argo-cd
          app.kubernetes.io/name: argocd-server-external
          app.kubernetes.io/part-of: argocd
          app.kubernetes.io/version: v2.8.4
      name: argocd-server-external
      spec:
      ports:
      - name: https
          port: 443
          protocol: TCP
          targetPort: 8080
          nodePort: 30443
      selector:
          app.kubernetes.io/instance: argo-cd
          app.kubernetes.io/name: argocd-server
      type: NodePort
      
        helm install <$resource_id> <$resource_id> \
            --namespace <$namespace> \
            --create-namespace \
            --version <$version> \
            --repo <$repo_url> \
            --values resource.values.yaml \
            --atomic
      crds:
          install: true
          keep: false
      global:
          revisionHistoryLimit: 3
          image:
              repository: m.daocloud.io/quay.io/argoproj/argocd
              imagePullPolicy: IfNotPresent
      redis:
          enabled: true
          image:
              repository: m.daocloud.io/docker.io/library/redis
          exporter:
              enabled: false
              image:
                  repository: m.daocloud.io/bitnami/redis-exporter
          metrics:
              enabled: false
      redis-ha:
          enabled: false
          image:
              repository: m.daocloud.io/docker.io/library/redis
          configmapTest:
              repository: m.daocloud.io/docker.io/koalaman/shellcheck
          haproxy:
              enabled: false
              image:
              repository: m.daocloud.io/docker.io/library/haproxy
          exporter:
              enabled: false
              image: m.daocloud.io/docker.io/oliver006/redis_exporter
      dex:
          enabled: true
          image:
              repository: m.daocloud.io/ghcr.io/dexidp/dex
      

    • debug resource

    kubectl -n <$namespace> describe <$resource_id>
    • logging resource
    kubectl -n <$namespace> logs -f <$resource_id>
    • port forwarding resource
    kubectl -n <$namespace> port-forward  <$resource_id> --address 0.0.0.0 8080:80 # local:pod
    • delete all resource under specific namespace
    kubectl delete all --all -n <$namespace>
    kubectl delete all --all --all-namespaces
    • delete error pods
    kubectl -n <$namespace> delete pods --field-selector status.phase=Failed
    • force delete
    kubectl -n <$namespace> delete pod <$resource_id> --force --grace-period=0
    • opening a Bash Shell inside a Pod
    kubectl -n <$namespace> exec -it <$resource_id> -- bash  
    • copy secret to another namespace
    kubectl -n <$namespaceA> get secret <$secret_name> -o json \
        | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid"])' \
        | kubectl -n <$namespaceB> apply -f -
    • copy secret to another name
    kubectl -n <$namespace> get secret <$old_secret_name> -o json | \
    jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid","ownerReferences","annotations","labels"]) | .metadata.name = "<$new_secret_name>"' | \
    kubectl apply -n <$namespace> -f -
    • delete all completed job
    kubectl delete jobs -n <$namespace> --field-selector status.successful=1 

    Nodes

    • add taint
    kubectl taint nodes <$node_ip> <key:value>
    kubectl taint nodes node1 dedicated:NoSchedule
    • remove taint
    kubectl remove taint
    kubectl taint nodes node1 dedicated:NoSchedule-
    • show info extract by json path
    kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

    Deploy

    • rollout show rollout history
    kubectl -n <$namespace> rollout history deploy/<$deploy_resource_id>

    undo rollout

    kubectl -n <$namespace> rollout undo deploy <$deploy_resource_id>  --to-revision=1

    Container 的子部分

    CheatShett

    type:
    1. remove specific image
    podman rmi <$image_id>
    1. remove all <none> images
    podman rmi `podamn images | grep  '<none>' | awk '{print $3}'`
    1. remove all stopped containers
    podman container prune
    1. remove all docker images not used
    podman image prune
    1. find ip address of a container
    podman inspect --format='{{.NetworkSettings.IPAddress}}' minio-server
    1. exec into container
    podman run -it <$container_id> /bin/bash
    1. run with environment
    podman run -d --replace 
        -p 18123:8123 -p 19000:9000 \
        --name clickhouse-server \
        -e ALLOW_EMPTY_PASSWORD=yes \
        --ulimit nofile=262144:262144 \
        quay.m.daocloud.io/kryptonite/clickhouse-docker-rootless:20.9.3.45 

    --ulimit nofile=262144:262144: 262144 is the maximum users process or for showing maximum user process limit for the logged-in user

    ulimit is admin access required Linux shell command which is used to see, set, or limit the resource usage of the current user. It is used to return the number of open file descriptors for each process. It is also used to set restrictions on the resources used by a process.

    1. login registry
    podman login --tls-verify=false --username=ascm-org-1710208820455 cr.registry.res.cloud.zhejianglab.com -p 'xxxx'
    1. tag image
    podman tag 76fdac66291c cr.registry.res.cloud.zhejianglab.com/ay-dev/datahub-s3-fits:1.0.0
    1. push image
    podman push cr.registry.res.cloud.zhejianglab.com/ay-dev/datahub-s3-fits:1.0.0
    1. remove specific image
    docker rmi <$image_id>
    1. remove all <none> images
    docker rmi `docker images | grep  '<none>' | awk '{print $3}'`
    1. remove all stopped containers
    docker container prune
    1. remove all docker images not used
    docker image prune
    1. find ip address of a container
    docker inspect --format='{{.NetworkSettings.IPAddress}}' minio-server
    1. exec into container
    docker exec -it <$container_id> /bin/bash
    1. run with environment
    docker run -d --replace -p 18123:8123 -p 19000:9000 --name clickhouse-server -e ALLOW_EMPTY_PASSWORD=yes --ulimit nofile=262144:262144 quay.m.daocloud.io/kryptonite/clickhouse-docker-rootless:20.9.3.45 

    --ulimit nofile=262144:262144: sssss

    1. copy file

      Copy a local file into container

      docker cp ./some_file CONTAINER:/work

      or copy files from container to local path

      docker cp CONTAINER:/var/logs/ /tmp/app_logs
    2. load a volume

    docker run --rm \
        --entrypoint bash \
        -v $PWD/data:/app:ro \
        -it docker.io/minio/mc:latest \
        -c "mc --insecure alias set minio https://oss-cn-hangzhou-zjy-d01-a.ops.cloud.zhejianglab.com/ g83B2sji1CbAfjQO 2h8NisFRELiwOn41iXc6sgufED1n1A \
            && mc --insecure ls minio/csst-prod/ \
            && mc --insecure mb --ignore-existing minio/csst-prod/crp-test \
            && mc --insecure cp /app/modify.pdf minio/csst-prod/crp-test/ \
            && mc --insecure ls --recursive minio/csst-prod/"

    Template 的子部分

    DevContainer Template

      DEV 的子部分

      Devpod

      Preliminary

      • Kubernetes has installed, if not check link
      • Devpod has installed, if not check link

      Get provider config

      # just copy ~/.kube/config

      for example, the original config

      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority: <$file_path>
          extensions:
          - extension:
              provider: minikube.sigs.k8s.io
              version: v1.33.0
            name: cluster_info
          server: https://<$minikube_ip>:8443
        name: minikube
      contexts:
      - context:
          cluster: minikube
          extensions:
          - extension:
              provider: minikube.sigs.k8s.io
              version: v1.33.0
            name: context_info
          namespace: default
          user: minikube
        name: minikube
      current-context: minikube
      kind: Config
      preferences: {}
      users:
      - name: minikube
        user:
          client-certificate: <$file_path>
          client-key: <$file_path>

      you need to rename clusters.cluster.certificate-authority, clusters.cluster.server, users.user.client-certificate, users.user.client-key.

      clusters.cluster.certificate-authority -> clusters.cluster.certificate-authority-data
      clusters.cluster.server -> clusters.cluster.server-data
      users.user.client-certificate -> users.user.client-certificate-data
      users.user.client-key -> users.user.client-key-data

      the data you paste after each key should be base64

      cat <$file_path> | base64

      then we should forward minikube port in your own pc

      #where you host minikube
      MACHINE_IP_ADDRESS=10.200.60.102
      USER=ayay
      MINIKUBE_IP_ADDRESS=$(ssh -o 'UserKnownHostsFile /dev/null' $USER@$MACHINE_IP_ADDRESS '$HOME/bin/minikube ip')
      ssh -o 'UserKnownHostsFile /dev/null' $USER@$MACHINE_IP_ADDRESS -L "*:8443:$MINIKUBE_IP_ADDRESS:8443" -N -f

      then, modified config file should be look like this:

      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: xxxxxxxxxxxxxx
          extensions:
          - extension:
              provider: minikube.sigs.k8s.io
              version: v1.33.0
            name: cluster_info
          server: https://127.0.0.1:8443 
        name: minikube
      contexts:
      - context:
          cluster: minikube
          extensions:
          - extension:
              provider: minikube.sigs.k8s.io
              version: v1.33.0
            name: context_info
          namespace: default
          user: minikube
        name: minikube
      current-context: minikube
      kind: Config
      preferences: {}
      users:
      - name: minikube
        user:
          client-certificate: xxxxxxxxxxxx
          client-key: xxxxxxxxxxxxxxxx

      Create workspace

      1. get git repo link
      2. choose appropriate provider
      3. choose ide type and version
      4. and go!

      Useful Command

      download kubectl binary

      MIRROR="files.m.daocloud.io/"
      VERSION=$(curl -L -s https://${MIRROR}dl.k8s.io/release/stable.txt)
      [ $(uname -m) = x86_64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/amd64/kubectl"
      [ $(uname -m) = aarch64 ] && curl -sSLo kubectl "https://${MIRROR}dl.k8s.io/release/${VERSION}/bin/linux/arm64/kubectl"
      chmod u+x kubectl
      mkdir -p ${HOME}/bin
      mv -f kubectl ${HOME}/bin

      Everything works fine.

      when you in pod, and using kubectl you should change clusters.cluster.server in ~/.kube/config to https://<$minikube_ip>:8443

      download argocd binary

      MIRROR="files.m.daocloud.io/"
      VERSION=v2.9.3
      [ $(uname -m) = x86_64 ] && curl -sSLo argocd "https://${MIRROR}github.com/argoproj/argo-cd/releases/download/${VERSION}/argocd-linux-amd64"
      [ $(uname -m) = aarch64 ] && curl -sSLo argocd "https://${MIRROR}github.com/argoproj/argo-cd/releases/download/${VERSION}/argocd-linux-arm64"
      chmod u+x argocd
      mkdir -p ${HOME}/bin
      mv -f argocd ${HOME}/bin

      exec into devpod

      kubectl -n devpod exec -it <$resource_id> -c devpod -- bin/bash

      add DNS item

      10.102.1.52 gitee.zhejianglab.com

      Deploy

        Operator 的子部分

        KubeBuilder

          Proxy 的子部分

          Daocloud

          1. install container tools

          systemctl stop firewalld && systemctl disable firewalld
          sudo dnf install -y podman
          podman run -d -P m.daocloud.io/docker.io/library/nginx

          Serverless 的子部分

          Knative 的子部分

          Eventing 的子部分

          Broker

            Plugin 的子部分

            Eventing Kafka Broker

              Kserve 的子部分

              Serving 的子部分

              Inference

                Generative

                  Canary Policy

                    Auto Scaling