kubernetes 问题，这里碰碰运气 (变相解决了)

eric_gao

车库里有个micro server， 内存很大，跑了好几个虚拟机，突发奇想，做个kubernetes cluster， 不想minikube，想做个真正的两节点cluster。

用的是最新1.27

master node 初始化一切正常

[student@control ~]$ kubectl get all -A
NAMESPACE      NAME                                  READY   STATUS    RESTARTS      AGE
kube-flannel   pod/kube-flannel-ds-txsfl             1/1     Running   0             23h
kube-system    pod/coredns-5d78c9869d-tp76m          1/1     Running   0             23h
kube-system    pod/coredns-5d78c9869d-tpgds          1/1     Running   0             23h
kube-system    pod/etcd-control                      1/1     Running   3 (23h ago)   23h
kube-system    pod/kube-apiserver-control            1/1     Running   3 (23h ago)   23h
kube-system    pod/kube-controller-manager-control   1/1     Running   3 (23h ago)   23h
kube-system    pod/kube-proxy-wfzlk                  1/1     Running   0             23h
kube-system    pod/kube-scheduler-control            1/1     Running   3 (23h ago)   23h

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  23h
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   23h

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   1         1         1       1            1           <none>                   23h
kube-system    daemonset.apps/kube-proxy        1         1         1       1            1           kubernetes.io/os=linux   23h

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           23h

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5d78c9869d   2         2         2       23h
[student@control ~]$

[student@control ~]$ kubectl get nodes
NAME      STATUS   ROLES           AGE   VERSION
control   Ready    control-plane   23h   v1.27.1
[student@control ~]$


但是节点1， join的时候，kubelet初始化失败， 提示failed to parse kubelet flag: unknown flag: --container-runtime


-- The start-up result is done.
Apr 26 20:58:14 node1 kubelet[82213]: E0426 20:58:14.003156   82213 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --cont>
Apr 26 20:58:14 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 26 20:58:14 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

貌似无解啊。 看了service 的定义，--container-runtime 是kubeadm join的时候动态生成的flag， 1.24-1.27 都取消了，目前只接受 --container-runtime=remote作为临时过度，  但是1.27 kubelet  join cluster的时候， 用的是 --container-runtime=docker

[student@node1 ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-04-26 21:01:28 AEST; 1s ago
     Docs: https://kubernetes.io/docs/
  Process: 82387 ExecStart=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=node1 --kubecon>
Main PID: 82387 (code=exited, status=1/FAILURE)

Apr 26 21:01:28 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 26 21:01:28 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
[student@node1 ~]$


想过downgrade node1的kubelet， 但是1.24-1.27 都是这个熊样，再往前，和master小版本相差太多，我记得好像worker node和master node最多可以相差2个小版本的。

想问问大神们有什么办法跳过这里？  谢谢， 谢谢，谢谢了。

eric_gao

具体细节

https://github.com/kubernetes/minikube/issues/16112

https://kubernetes.io/blog/2023/ ... n-kubernetes-v1-27/

SLCOW

Removal of --container-runtime command line argument
The kubelet accepts a deprecated command line argument, --container-runtime, and the only valid value would be remote after dockershim code is removed. Kubernetes v1.27 will remove that argument, which has been deprecated since the v1.24 release.

SLCOW

试一下 /var/lib/kubelet/kubeadm-flags.env ， which contains a list of flags to pass to the kubelet when it starts.

eric_gao

SLCOW 发表于 2023-4-26 21:28
试一下 /var/lib/kubelet/kubeadm-flags.env ， which contains a list of flags to pass to the kubelet w ...

试过了以下两个

/var/lib/kubelet/kubeadm-flags.env

/etc/sysconfig/kubelet

也重新安装了kubelet， 但是kubeadm join，一初始化， 还是 --container-runtime=docker，  气的半死。

neohope

你的container runtime跑的啥？  试试containerd

When Docker runtime support is removed in a future release (currently planned for the 1.22 release in late 2021) of Kubernetes it will no longer be supported and you will need to switch to one of the other compliant container runtimes, like。containerd or CRI-O

eric_gao

neohope 发表于 2023-4-27 09:50
你的container runtime跑的啥？  试试containerd

When Docker runtime support is removed in a future re ...

我用的就是containerd

间风

不知道你是什么系统，我在 ubuntu server 22.04.1上测试毫无问题
Master Node:

1. #!/bin/bash
   
2. 
   
3. # Synchronize the package index files and install the packages below
   
4. sudo apt-get update
   
5. 
   
6. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
   
7.         ca-certificates \
   
8.         curl \
   
9.         gnupg \
   
10.         lsb-release
    
11. 
    
12. sudo mkdir -p /etc/apt/keyrings
    
13. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    
14. echo \
    
15.         "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    
16.         $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    
17. 
    
18. 
    
19. sudo apt-get update
    
20. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y  containerd.io
    
21. 
    
22. # setup containerd
    
23. sudo mkdir -p /etc/containerd
    
24. containerd config default | sudo tee /etc/containerd/config.toml
    
25. sudo sed -i 's/SystemdCgroup\ =\ false/SystemdCgroup\ =\ true/g' /etc/containerd/config.toml
    
26. sudo systemctl restart containerd
    
27. 
    
28. # install kubernetes
    
29. sudo apt-get update
    
30. sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    
31. echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    
32. 
    
33. sudo apt-get update
    
34. k8sversion=1.27.1-00
    
35. echo "sudo apt-get install -y kubelet=$k8sversion kubeadm=$k8sversion kubectl=$k8sversion"
    
36. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y kubelet=${k8sversion} kubeadm=${k8sversion} kubectl=${k8sversion}
    
37. sudo apt-mark hold kubeadm kubelet kubectl
    
38. 
    
39. # --pod-network-cidr=10.244.0.0/16 match https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
    
40. # If you use custom podCIDR (not 10.244.0.0/16) you first need to download the above manifest and modify the network to match your one.
    
41. sudo kubeadm init --pod-network-cidr=10.244.0.0/16
    
42. 
    
43. mkdir -p $HOME/.kube
    
44. sudo cp -rf /etc/kubernetes/admin.conf $HOME/.kube/config
    
45. sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
46. 
    
47. #CNI
    
48. kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
    
49. 
    
50. #kubectl taint nodes --all node-role.kubernetes.io/master-
    
51. # for k8s 1.24 and above
    
52. kubectl taint nodes --all node-role.kubernetes.io/control-plane-
    
53. 
    
54. kubectl --namespace=kube-system create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
    
55. kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.11.1/cert-manager.yaml
    
56. sudo openssl dhparam -out dh-param 2048
    
57. kubectl create --namespace=kube-system secret generic tls-dhparam --from-file=./dh-param
    
58. 
    
59. sudo crictl config --set runtime-endpoint=unix:///run/containerd/containerd.sock

复制代码

# 略去 copy .kube 和 chown

Slave Node:

1. #Forwarding IPv4 and letting iptables see bridged traffic
   
2. cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
   
3. overlay
   
4. br_netfilter
   
5. EOF
   
6. 
   
7. sudo modprobe overlay
   
8. sudo modprobe br_netfilter
   
9. 
   
10. # sysctl params required by setup, params persist across reboots
    
11. cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    
12. net.bridge.bridge-nf-call-iptables  = 1
    
13. net.bridge.bridge-nf-call-ip6tables = 1
    
14. net.ipv4.ip_forward                 = 1
    
15. EOF
    
16. 
    
17. # Apply sysctl params without reboot
    
18. sudo sysctl --system
    
19. 
    
20. # verify that the br_netfilter, overlay modules are loaded by running below instruction
    
21. lsmod | grep br_netfilter
    
22. lsmod | grep overlay
    
23. 
    
24. 
    
25. 
    
26. #!/bin/bash
    
27. 
    
28. # Synchronize the package index files and install the packages below
    
29. sudo apt-get update
    
30. 
    
31. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y \
    
32.         ca-certificates \
    
33.         curl \
    
34.         gnupg \
    
35.         lsb-release
    
36. 
    
37. sudo mkdir -p /etc/apt/keyrings
    
38. curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
    
39. echo \
    
40.         "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
    
41.         $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    
42. 
    
43. 
    
44. sudo apt-get update
    
45. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y  containerd.io
    
46. 
    
47. # setup containerd
    
48. sudo mkdir -p /etc/containerd
    
49. containerd config default | sudo tee /etc/containerd/config.toml
    
50. sudo sed -i 's/SystemdCgroup\ =\ false/SystemdCgroup\ =\ true/g' /etc/containerd/config.toml
    
51. sudo systemctl restart containerd
    
52. 
    
53. # install kubernetes
    
54. sudo apt-get update
    
55. sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    
56. echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    
57. 
    
58. sudo apt-get update
    
59. k8sversion=1.27.1-00
    
60. echo "sudo apt-get install -y kubelet=$k8sversion kubeadm=$k8sversion kubectl=$k8sversion"
    
61. sudo DEBIAN_FRONTEND=noninteractive apt-get install -y kubelet=${k8sversion} kubeadm=${k8sversion} kubectl=${k8sversion}
    
62. sudo apt-mark hold kubeadm kubelet kubectl
    

复制代码

Join cluster

1. # get join command from master node
   
2. sudo kubeadm token create --print-join-command
   
3. # transfer ~/.kube  to slave node
   
4. scp -r ~/.kube slave:~/

复制代码

# run on slave node

1. kubeadm join x.x.x.x:6443 --token  xxxxxx --discovery-token-ca-cert-hash sha256:xxxxxx

复制代码

间风

1. ****Identify the cgroup version on Linux Nodes****

1. stat -fc %T /sys/fs/cgroup/

复制代码

For cgroup v2, the output is `cgroup2fs`.

For cgroup v1, the output is `tmpfs.`

Note: ubuntu 22.04 is cgroup v2

eric_gao

间风 发表于 2023-4-27 13:51
1. ****Identify the cgroup version on Linux Nodes****

[root@node1 kubelet.service.d]# stat -fc %T /sys/fs/cgroup/
tmpfs
[root@node1 kubelet.service.d]#

下一步是啥呢，老大？

eric_gao

间风 发表于 2023-4-27 13:51
1. ****Identify the cgroup version on Linux Nodes****

我的control node 是 redhat 8
node1 是centos 8

eric_gao

间风 发表于 2023-4-27 13:51
1. ****Identify the cgroup version on Linux Nodes****

能发一下你的worker node  "systemctl status kubelet" ? 我想看看你的kubelet process是什么

[student@node1 ~]$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2023-04-27 16:08:32 AEST; 707ms ago
     Docs: https://kubernetes.io/docs/
  Process: 27838 ExecStart=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=node1 --kubeconfig=/etc/kubernetes/ku>
Main PID: 27838 (code=exited, status=1/FAILURE)

Apr 27 16:08:32 node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 27 16:08:32 node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
[student@node1 ~]$

我这里试着改文件，改service 文件，hard code ExecStart 都不行，就是--container-runtime=docker 搞不掉，一直在。

间风

eric_gao 发表于 2023-4-27 16:04
[root@node1 kubelet.service.d]# stat -fc %T /sys/fs/cgroup/
tmpfs
[root@node1 kubelet.service.d]#

https://kubernetes.io/docs/setup ... untimes/#containerd

Configuring the systemd cgroup driver
To use the systemd cgroup driver in /etc/containerd/config.toml with runc, set

1. [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
   
2.   ...
   
3.   [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
   
4.     SystemdCgroup = true

复制代码

The systemd cgroup driver is recommended if you use cgroup v2.
你的既然不是cgroupv2 那么保持 SystemdCgroup=false