kubernetes 问题，这里碰碰运气 (变相解决了)

eric_gao

车库里有个micro server， 内存很大，跑了好几个虚拟机，突发奇想，做个kubernetes cluster， 不想minikube，想做个真正的两节点cluster。

用的是最新1.27

master node 初始化一切正常

[student@control ~]$ kubectl get all -A
NAMESPACE      NAME                                  READY   STATUS    RESTARTS      AGE
kube-flannel   pod/kube-flannel-ds-txsfl             1/1     Running   0             23h
kube-system    pod/coredns-5d78c9869d-tp76m          1/1     Running   0             23h
kube-system    pod/coredns-5d78c9869d-tpgds          1/1     Running   0             23h
kube-system    pod/etcd-control                      1/1     Running   3 (23h ago)   23h
kube-system    pod/kube-apiserver-control            1/1     Running   3 (23h ago)   23h
kube-system    pod/kube-controller-manager-control   1/1     Running   3 (23h ago)   23h
kube-system    pod/kube-proxy-wfzlk                  1/1     Running   0             23h
kube-system    pod/kube-scheduler-control            1/1     Running   3 (23h ago)   23h

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  23h
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   23h

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   1         1         1       1            1           <none>                   23h
kube-system    daemonset.apps/kube-proxy        1         1         1       1            1           kubernetes.io/os=linux   23h

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           23h

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5d78c9869d   2         2         2       23h
[student@control ~]$

[student@control ~]$ kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
control   Ready    control-plane   23h   v1.27.1
[student@control ~]$


但是节点1， join的时候，kubelet初始化失败， 提示failed to parse kubelet flag: unknown flag: --container-runtime


-- The start-up result is done.
Apr 26 20:58:14 node1 kubelet[82213]: E0426 20:58:14.003156   82213 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --cont>
Apr 26 20:58:14 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 26 20:58:14 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

貌似无解啊。 看了service 的定义，--container-runtime 是kubeadm join的时候动态生成的flag， 1.24-1.27 都取消了，目前只接受 --container-runtime=remote作为临时过度，  但是1.27 kubelet  join cluster的时候， 用的是 --container-runtime=docker

[student@node1 ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-04-26 21:01:28 AEST; 1s ago
     Docs: https://kubernetes.io/docs/
  Process: 82387 ExecStart=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=node1 --kubecon>
Main PID: 82387 (code=exited, status=1/FAILURE)

Apr 26 21:01:28 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 26 21:01:28 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
[student@node1 ~]$


想过downgrade node1的kubelet， 但是1.24-1.27 都是这个熊样，再往前，和master小版本相差太多，我记得好像worker node和master node最多可以相差2个小版本的。

想问问大神们有什么办法跳过这里？  谢谢， 谢谢，谢谢了。

eric_gao

具体细节

https://github.com/kubernetes/minikube/issues/16112

https://kubernetes.io/blog/2023/ ... n-kubernetes-v1-27/

SLCOW

Removal of --container-runtime command line argument
The kubelet accepts a deprecated command line argument, --container-runtime, and the only valid value would be remote after dockershim code is removed. Kubernetes v1.27 will remove that argument, which has been deprecated since the v1.24 release.

SLCOW

试一下 /var/lib/kubelet/kubeadm-flags.env ， which contains a list of flags to pass to the kubelet when it starts.

eric_gao

SLCOW 发表于 2023-4-26 21:28
试一下 /var/lib/kubelet/kubeadm-flags.env ， which contains a list of flags to pass to the kubelet w ...

试过了以下两个

/var/lib/kubelet/kubeadm-flags.env

/etc/sysconfig/kubelet

也重新安装了kubelet， 但是kubeadm join，一初始化， 还是 --container-runtime=docker，  气的半死。

neohope

你的container runtime跑的啥？  试试containerd

When Docker runtime support is removed in a future release (currently planned for the 1.22 release in late 2021) of Kubernetes it will no longer be supported and you will need to switch to one of the other compliant container runtimes, like。containerd or CRI-O

eric_gao

neohope 发表于 2023-4-27 09:50
你的container runtime跑的啥？  试试containerd

When Docker runtime support is removed in a future re ...

我用的就是containerd

间风

不知道你是什么系统，我在 ubuntu server 22.04.1上测试毫无问题
Master Node:

1. #!/bin/bash
   
2. 
   
3. # Synchronize the package index files and install the packages below
   
4. sudo apt-get update
   
5. 
   
6. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
   
7.         ca-certificates \
   
8.         curl \
   
9.         gnupg \
   
10.         lsb-release
    
11. 
    
12. sudo mkdir -p /etc/apt/keyrings
    
13. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    
14. echo \
    
15.         "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    
16.         $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    
17. 
    
18. 
    
19. sudo apt-get update
    
20. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y  containerd.io
    
21. 
    
22. # setup containerd
    
23. sudo mkdir -p /etc/containerd
    
24. containerd config default | sudo tee /etc/containerd/config.toml
    
25. sudo sed -i 's/SystemdCgroup\ =\ false/SystemdCgroup\ =\ true/g' /etc/containerd/config.toml
    
26. sudo systemctl restart containerd
    
27. 
    
28. # install kubernetes
    
29. sudo apt-get update
    
30. sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    
31. echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    
32. 
    
33. sudo apt-get update
    
34. k8sversion=1.27.1-00
    
35. echo "sudo apt-get install -y kubelet=$k8sversion kubeadm=$k8sversion kubectl=$k8sversion"
    
36. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y kubelet=${k8sversion} kubeadm=${k8sversion} kubectl=${k8sversion}
    
37. sudo apt-mark hold kubeadm kubelet kubectl
    
38. 
    
39. # --pod-network-cidr=10.244.0.0/16 match https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
    
40. # If you use custom podCIDR (not 10.244.0.0/16) you first need to download the above manifest and modify the network to match your one.
    
41. sudo kubeadm init --pod-network-cidr=10.244.0.0/16
    
42. 
    
43. mkdir -p $HOME/.kube
    
44. sudo cp -rf /etc/kubernetes/admin.conf $HOME/.kube/config
    
45. sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
46. 
    
47. #CNI
    
48. kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
    
49. 
    
50. #kubectl taint nodes --all node-role.kubernetes.io/master-
    
51. # for k8s 1.24 and above
    
52. kubectl taint nodes --all node-role.kubernetes.io/control-plane-
    
53. 
    
54. kubectl --namespace=kube-system create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
    
55. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.11.1/cert-manager.yaml
    
56. sudo openssl dhparam -out dh-param 2048
    
57. kubectl create --namespace=kube-system secret generic tls-dhparam --from-file=./dh-param
    
58. 
    
59. sudo crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

复制代码

# 略去 copy .kube 和 chown

Slave Node:

1. #Forwarding IPv4 and letting iptables see bridged traffic
   
2. cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
   
3. overlay
   
4. br_netfilter
   
5. EOF
   
6. 
   
7. sudo modprobe overlay
   
8. sudo modprobe br_netfilter
   
9. 
   
10. # sysctl params required by setup, params persist across reboots
    
11. cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    
12. net.bridge.bridge-nf-call-iptables  = 1
    
13. net.bridge.bridge-nf-call-ip6tables = 1
    
14. net.ipv4.ip_forward                 = 1
    
15. EOF
    
16. 
    
17. # Apply sysctl params without reboot
    
18. sudo sysctl --system
    
19. 
    
20. # verify that the br_netfilter, overlay modules are loaded by running below instruction
    
21. lsmod | grep br_netfilter
    
22. lsmod | grep overlay
    
23. 
    
24. 
    
25. 
    
26. #!/bin/bash
    
27. 
    
28. # Synchronize the package index files and install the packages below
    
29. sudo apt-get update
    
30. 
    
31. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
    
32.         ca-certificates \
    
33.         curl \
    
34.         gnupg \
    
35.         lsb-release
    
36. 
    
37. sudo mkdir -p /etc/apt/keyrings
    
38. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    
39. echo \
    
40.         "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    
41.         $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    
42. 
    
43. 
    
44. sudo apt-get update
    
45. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y  containerd.io
    
46. 
    
47. # setup containerd
    
48. sudo mkdir -p /etc/containerd
    
49. containerd config default | sudo tee /etc/containerd/config.toml
    
50. sudo sed -i 's/SystemdCgroup\ =\ false/SystemdCgroup\ =\ true/g' /etc/containerd/config.toml
    
51. sudo systemctl restart containerd
    
52. 
    
53. # install kubernetes
    
54. sudo apt-get update
    
55. sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    
56. echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    
57. 
    
58. sudo apt-get update
    
59. k8sversion=1.27.1-00
    
60. echo "sudo apt-get install -y kubelet=$k8sversion kubeadm=$k8sversion kubectl=$k8sversion"
    
61. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y kubelet=${k8sversion} kubeadm=${k8sversion} kubectl=${k8sversion}
    
62. sudo apt-mark hold kubeadm kubelet kubectl
    

复制代码

Join cluster

1. # get join command from master node
   
2. sudo kubeadm token create --print-join-command
   
3. # transfer ~/.kube  to slave node
   
4. scp -r ~/.kube slave:~/

复制代码

# run on slave node

1. kubeadm join x.x.x.x:6443 --token  xxxxxx --discovery-token-ca-cert-hash sha256:xxxxxx

复制代码

间风

1. ****Identify the cgroup version on Linux Nodes****

1. stat -fc %T /sys/fs/cgroup/

复制代码

For cgroup v2, the output is `cgroup2fs`.

For cgroup v1, the output is `tmpfs.`

Note: ubuntu 22.04 is cgroup v2

eric_gao

间风 发表于 2023-4-27 13:51
1. ****Identify the cgroup version on Linux Nodes****

[root@node1 kubelet.service.d]# stat -fc %T /sys/fs/cgroup/
tmpfs
[root@node1 kubelet.service.d]#

下一步是啥呢，老大？

eric_gao

间风 发表于 2023-4-27 13:51
1. ****Identify the cgroup version on Linux Nodes****

我的control node 是 redhat 8
node1 是centos 8

eric_gao

间风 发表于 2023-4-27 13:51
1. ****Identify the cgroup version on Linux Nodes****

能发一下你的worker node  "systemctl status kubelet" ? 我想看看你的kubelet process是什么

[student@node1 ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2023-04-27 16:08:32 AEST; 707ms ago
     Docs: https://kubernetes.io/docs/
  Process: 27838 ExecStart=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=node1 --kubeconfig=/etc/kubernetes/ku>
Main PID: 27838 (code=exited, status=1/FAILURE)

Apr 27 16:08:32 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 27 16:08:32 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
[student@node1 ~]$

我这里试着改文件，改service 文件，hard code ExecStart 都不行，就是--container-runtime=docker 搞不掉，一直在。

间风

eric_gao 发表于 2023-4-27 16:04
[root@node1 kubelet.service.d]# stat -fc %T /sys/fs/cgroup/
tmpfs
[root@node1 kubelet.service.d]#

https://kubernetes.io/docs/setup ... untimes/#containerd

Configuring the systemd cgroup driver
To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set

1. [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
   
2.   ...
   
3.   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
   
4.     SystemdCgroup = true

复制代码

The systemd cgroup driver is recommended if you use cgroup v2.
你的既然不是cgroupv2 那么保持 SystemdCgroup=false

间风

eric_gao 发表于 2023-4-27 16:11
能发一下你的worker node  "systemctl status kubelet" ? 我想看看你的kubelet process是什么

[student@ ...

Master Node:

1. ● kubelet.service - kubelet: The Kubernetes Node Agent
   
2.      Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   
3.     Drop-In: /etc/systemd/system/kubelet.service.d
   
4.              └─10-kubeadm.conf
   
5.      Active: active (running) since Wed 2023-04-19 21:33:36 UTC; 1 week 0 days ago
   
6.        Docs: https://kubernetes.io/docs/home/
   
7.    Main PID: 2404 (kubelet)
   
8.       Tasks: 25 (limit: 28711)
   
9.      Memory: 96.8M
   
10.         CPU: 10h 43min 21.709s
    
11.      CGroup: /system.slice/kubelet.service
    
12.              └─2404 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9
    

复制代码

Slave Node:

1. ● kubelet.service - kubelet: The Kubernetes Node Agent
   
2.      Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   
3.     Drop-In: /etc/systemd/system/kubelet.service.d
   
4.              └─10-kubeadm.conf
   
5.      Active: active (running) since Thu 2023-04-27 03:21:36 UTC; 6h ago
   
6.        Docs: https://kubernetes.io/docs/home/
   
7.    Main PID: 4321 (kubelet)
   
8.       Tasks: 17 (limit: 28715)
   
9.      Memory: 39.3M
   
10.         CPU: 5min 16.130s
    
11.      CGroup: /system.slice/kubelet.service
    
12.              └─4321 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9

复制代码

间风

1. cat /var/lib/kubelet/kubeadm-flags.env
   

复制代码

1. KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"

复制代码

eric_gao

间风 发表于 2023-4-27 19:47

我的kubeadm-flags.env 也是你这样，
[root@node1 kubelet.service.d]# cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS="--container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9"
[root@node1 kubelet.service.d]#

还试着hard code service 启动命令，确保哪里也没有--container-runtime=docker，但是kubelet 服务里面就是有这项， 应该是kubeadm join的时候，动态生成的flags，真是见了鬼

[root@node1 kubelet.service.d]# pwd
/usr/lib/systemd/system/kubelet.service.d

[root@node1 kubelet.service.d]# cat 10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
#ExecStart=
#ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
ExecStart=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9
[root@node1 kubelet.service.d]#

飞翔翼

kubernetes cluster用rancher建很方便，我自己的虚拟机上用过rancher后就不想自己手工做了。

eric_gao

间风 发表于 2023-4-27 17:19
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

Configurin ...

问题还没到cgroup，我已经设置了用systemd， 但是这应该只会影响后续deployment的cpu， 内存限制啥的， 我这还没到这里呢。

现在worker node的kubelet，在kubeadm join的时候，初始化失败，得想法把这关过去。

eric_gao

飞翔翼 发表于 2023-4-27 19:54
kubernetes cluster用rancher建很方便，我自己的虚拟机上用过rancher后就不想自己手工做了。 ...

rancher是个什么东东，还没用过。 我这是一步一步手工建的。

飞翔翼

eric_gao 发表于 2023-4-27 19:55
rancher是个什么东东，还没用过。 我这是一步一步手工建的。

K8S管理器呀，你放狗搜一下就知道了。

间风

eric_gao 发表于 2023-4-27 19:55
问题还没到cgroup，我已经设置了用systemd， 但是这应该只会影响后续deployment的cpu， 内存限制啥的，  ...

检查下

1. kubectl edit node

复制代码

kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock

eric_gao

间风 发表于 2023-4-27 20:25
检查下

apiVersion: v1
kind: Node
metadata:
  annotations:
    flannel.alpha.coreos.com/backend-data: '{"VNI":1,"VtepMAC":"ea:dd:9d:a2:d7:74"}'
    flannel.alpha.coreos.com/backend-type: vxlan
    flannel.alpha.coreos.com/kube-subnet-manager: "true"
    flannel.alpha.coreos.com/public-ip: 192.168.1.223
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2023-04-25T11:40:18Z"
  labels:

间风

@eric_gao
搭了一个跟你一样的环境，master node: rhel 8.7 slave node: centos 8

both disable selinux, firewalld, swap

全部执行以下命令

1. cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
   
2. overlay
   
3. br_netfilter
   
4. EOF
   
5. 
   
6. sudo modprobe overlay
   
7. sudo modprobe br_netfilter
   
8. 
   
9. # sysctl params required by setup, params persist across reboots
   
10. cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    
11. net.bridge.bridge-nf-call-iptables  = 1
    
12. net.bridge.bridge-nf-call-ip6tables = 1
    
13. net.ipv4.ip_forward                 = 1
    
14. EOF
    
15. 
    
16. # Apply sysctl params without reboot
    
17. sudo sysctl --system
    
18. 
    
19. # verify that the br_netfilter, overlay modules are loaded by running below instruction
    
20. lsmod | grep br_netfilter
    
21. lsmod | grep overlay

复制代码

master node:
手动安装containerd, runc, crictl

1. # Installing containerd
   
2. containerd_version=1.6.20
   
3. wget https://github.com/containerd/containerd/releases/download/v$containerd_version/containerd-$containerd_version-linux-amd64.tar.gz
   
4. # extract to /usr/local/bin
   
5. sudo tar Cxzvf /usr/local containerd-$containerd_version-linux-amd64.tar.gz
   
6. rm containerd-$containerd_version-linux-amd64.tar.gz
   
7. # service
   
8. wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
   
9. sudo mv containerd.service /etc/systemd/system/containerd.service
   
10. # create config.toml
    
11. sudo mkdir -p /etc/containerd/
    
12. containerd config default|sudo tee /etc/containerd/config.toml
    
13. 
    
14. # start containerd
    
15. sudo systemctl daemon-reload
    
16. sudo systemctl enable --now containerd
    
17. 
    
18. # Installing runc
    
19. runc_version=v1.1.5
    
20. wget https://github.com/opencontainers/runc/releases/download/$runc_version/runc.amd64
    
21. sudo install -m 755 runc.amd64 /usr/local/sbin/runc
    
22. rm runc.amd64
    
23. 
    
24. # Installing CNI
    
25. cni_version=v1.2.0
    
26. wget https://github.com/containernetworking/plugins/releases/download/$cni_version/cni-plugins-linux-amd64-$cni_version.tgz
    
27. sudo mkdir -p /opt/cni/bin
    
28. sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-$cni_version.tgz
    
29. rm cni-plugins-linux-amd64-$cni_version.tgz
    
30. 
    
31. # crictl endpoint
    
32. VERSION="v1.26.1"
    
33. wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
    
34. sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
    
35. rm -f crictl-$VERSION-linux-amd64.tar.gz
    
36. crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

复制代码

Install K8S
https://kubernetes.io/docs/setup ... dm/install-kubeadm/

1. cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
   
2. [kubernetes]
   
3. name=Kubernetes
   
4. baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
   
5. enabled=1
   
6. gpgcheck=1
   
7. gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
   
8. exclude=kubelet kubeadm kubectl
   
9. EOF
   
10. 
    
11. 
    
12. sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
    
13. 
    
14. sudo systemctl enable --now kubelet

复制代码

初始化

1. sudo kubeadm init --pod-network-cidr=10.244.0.0/16
   
2. mkdir -p $HOME/.kube
   
3. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
   
4. sudo chown $(id -u):$(id -g) $HOME/.kube/config
   
5. 
   
6. # deploy network
   
7. kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
   
8. 
   
9. # taint control panel node
   
10. kubectl taint nodes --all node-role.kubernetes.io/control-plane-

复制代码

Slave Node:
重复 手动安装containerd, runc, crictl 部分
重复 Install K8S 部分
安装 tc for k8s join pre check on slave node

1. dnf install -y iproute-tc
   

复制代码

scp ~/.kube from master node to slave node
run

1. kube join

复制代码

from slave node

eric_gao

间风 发表于 2023-4-28 23:10
@eric_gao
搭了一个跟你一样的环境，master node: rhel 8.7 slave node: centos 8

谢谢了， 又手工装了一遍runtime和k8s， join的时候，还是如此 --container-runtime=docker 一直在。 放弃了。

今天下午又临时装了个ubuntu，这个作为workder node加入，一点问题都没有。

[student@control ~]$ kubectl get nodes
NAME      STATUS   ROLES           AGE     VERSION
control   Ready    control-plane   3d19h   v1.27.1
node5     Ready    <none>          44s     v1.27.1
[student@control ~]$
[student@control ~]$


这下方便了，做了个虚拟机备份， 有个k8s在手，接下来玩一玩api gateway， service mesh，方便多了。

ypqfyf

虽然但是，我觉得kind也挺方便的，也可以多节点，想玩啥都可以