Investigation ideas in the K8S cluster encounter ETCD cluster faults

Create an instance in the k8s cluster once and find that the etcd cluster status has a connection failure, resulting in Failed to create instance. So I checked the cause.

Source of the problem

Here is the health status of etcd cluster:

It is obvious that there is a problem with etcd node 03 .

At this time, restart the etcd service on node 03 as follows:

并没有成功启动服务，可以看到提示信息：member c2c5804bd87e2884 has already been bootstrapped

查看资料说是：
One of the member was bootstrapped via discovery service. You must remove the previous data-dir to clean up the member information. Or the member will ignore the new configuration and start with the old configuration. That is why you see the mismatch.
大概意思：
其中一个成员是通过discovery service引导的。必须删除以前的数据目录来清理成员信息。否则成员将忽略新配置，使用旧配置。这就是为什么你看到了不匹配。
看到了这里，问题所在也就很明确了，启动失败的原因在于data-dir （/var/lib/etcd/default.etcd）中记录的信息与 etcd启动的选项所标识的信息不太匹配造成的。

问题解决

第一种方式我们可以通过修改启动参数解决这类错误。既然 data-dir 中已经记录信息，我们就没必要在启动项中加入多于配置。具体修改--initial-cluster-state参数：

我们将 --initial-cluster-state=new 修改成  --initial-cluster-state=existing，再次重新启动就ok了。

第二种方式删除所有etcd节点的 data-dir 文件（不删也行），重启各个节点的etcd服务，这个时候，每个节点的data-dir的数据都会被更新，就不会有以上故障了。

第三种方式是复制其他节点的data-dir中的内容，以此为基础上以 --force-new-cluster 的形式强行拉起一个，然后以添加新成员的方式恢复这个集群。

这是目前的几种解决办法

1 2 3 4 5 6 7 8 9 10 11

[[email protected] ~] # cd /opt/kubernetes/ssl/ [[email protected] ssl] # /opt/kubernetes/bin/etcdctl \ > --ca- file =ca.pem --cert- file =server.pem --key- file =server-key.pem \ > --endpoints= "https://10.0.0.99:2379,https://10.0.0.100:2379,https://10.0.0.111:2379" \ > cluster-health member 1bd4d12de986e887 is healthy: got healthy result from https: //10 .0.0.99:2379 member 45396926a395958b is healthy: got healthy result from https: //10 .0.0.100:2379 failed to check the health of member c2c5804bd87e2884 on https: //10 .0.0.111:2379: Get https: //10 .0.0.111:2379 /health : net /http : TLS handshake timeout member c2c5804bd87e2884 is unreachable: [https: //10 .0.0.111 :2379] are all unreachable cluster is healthy [[email protected] ssl] #

1

2

3

4

5

6

7

8

9

10

11

[[email protected] ~] # cd /opt/kubernetes/ssl/

[[email protected] ssl] # /opt/kubernetes/bin/etcdctl \

> --ca- file =ca.pem --cert- file =server.pem --key- file =server-key.pem \

> --endpoints= "https://10.0.0.99:2379,https://10.0.0.100:2379,https://10.0.0.111:2379" \

> cluster-health

member 1bd4d12de986e887 is healthy: got healthy result from https: //10 .0.0.99:2379

member 45396926a395958b is healthy: got healthy result from https: //10 .0.0.100:2379

failed to check the health of member c2c5804bd87e2884 on https: //10 .0.0.111:2379: Get https: //10 .0.0.111:2379 /health : net /http : TLS handshake timeout

member c2c5804bd87e2884 is unreachable: [https: //10 .0.0.111:2379] are all unreachable

cluster is healthy

[[email protected] ssl] #

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

[[email protected] ~] # systemctl restart etcd Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details. [[email protected] ~] # journalctl -xe Mar 24 22:24:32 docker03 etcd[1895]: setting maximum number of CPUs to 1, total number of available CPUs is 1 Mar 24 22:24:32 docker03 etcd[1895]: the server is already initialized as member before, starting as etcd member... Mar 24 22:24:32 docker03 etcd[1895]: peerTLS: cert = /opt/kubernetes/ssl/server .pem, key = /opt/kubernetes/ssl/server-key .pem, ca = , trusted-ca = /opt/kubernetes/ssl Mar 24 22:24:32 docker03 etcd[1895]: listening for peers on https: //10 .0.0.111:2380 Mar 24 22:24:32 docker03 etcd[1895]: The scheme of client url http: //127 .0.0.1:2379 is HTTP while peer key /cert files are presented. Ignored key /cert files. Mar 24 22:24:32 docker03 etcd[1895]: listening for client requests on 127.0.0.1:2379 Mar 24 22:24:32 docker03 etcd[1895]: listening for client requests on 10.0.0.111:2379 Mar 24 22:24:32 docker03 etcd[1895]: member c2c5804bd87e2884 has already been bootstrapped Mar 24 22:24:32 docker03 systemd[1]: etcd.service: main process exited, code=exited, status=1 /FAILURE Mar 24 22:24:32 docker03 systemd[1]: Failed to start Etcd Server. -- Subject: Unit etcd.service has failed -- Defined-By: systemd -- Support: http: //lists .freedesktop.org /mailman/listinfo/systemd-devel -- -- Unit etcd.service has failed. -- -- The result is failed. Mar 24 22:24:32 docker03 systemd[1]: Unit etcd.service entered failed state. Mar 24 22:24:32 docker03 systemd[1]: etcd. service failed. Mar 24 22:24:33 docker03 systemd[1]: etcd.service holdoff time over, scheduling restart. Mar 24 22:24:33 docker03 systemd[1]: start request repeated too quickly for etcd.service Mar 24 22:24:33 docker03 systemd[1]: Failed to start Etcd Server. -- Subject: Unit etcd.service has failed -- Defined-By: systemd -- Support: http: //lists .freedesktop.org /mailman/listinfo/systemd-devel -- -- Unit etcd.service has failed. -- -- The result is failed. Mar 24 22:24:33 docker03 systemd[1]: Unit etcd.service entered failed state. Mar 24 22:24:33 docker03 systemd [1]: etcd.service failed.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

-- Support: http: //lists .freedesktop.org /mailman/listinfo/systemd-devel

--

-- Unit etcd.service has failed.

--

-- The result is failed.

Mar 24 22:24:32 docker03 systemd[1]: Unit etcd.service entered failed state.

Mar 24 22:24:32 docker03 systemd[1]: etcd.service failed.

[[email protected] ~] # systemctl restart etcd

Job for etcd.service failed bec ause the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.

[[email protected] ~] # journalctl -xe

Mar 24 22:24:32 docker03 etcd[1895]: setting maximum number of CPUs to 1, total number of available CPUs is 1

Mar 24 22:24:32 docker03 etcd[1895]: the server is already initialized as member before, starting as etcd member...

Mar 24 22:24:32 docker03 etcd[1895]: peerTLS: cert = /opt/kubernetes/ssl/server .pem, key = /opt/kubernetes/ssl/server-key .pem, ca = , trusted-ca = /opt/kubernetes/ssl

Mar 24 22:24:32 docker03 etcd[1895]: listening for peers on https: //10 .0.0.111:2380

Mar 24 22:24:32 docker03 etcd[1895]: The scheme of client url http: //127 .0.0.1:2379 is HTTP while peer key /cert files are presented. Ignored key /cert files.

Mar 24 22:24:32 docker03 etcd[1895]: listening for client requests on 127.0.0.1:2379

Mar 24 22:24:32 docker03 etcd[1895]: listening for client requests on 10.0.0.111:2379

Mar 24 22:24:32 docker03 etcd[1895]: member c2c5804bd87e2884 has already been bootstrapped

Mar 24 22:24:32 docker03 systemd[1]: etcd.service: main process exited, code=exited, status=1 /FAILURE

Mar 24 22:24:32 docker03 systemd[1]: Failed to start Etcd Server.

-- Subject: Unit etcd.service has failed

-- Defined-By: systemd

-- Support: http: //lists .freedesktop.org /mailman/listinfo/systemd-devel

--

-- Unit etcd.service has failed.

--

-- The result is failed.

Mar 24 22:24:32 docker03 systemd[1]: Unit etcd.service entered failed state.

Mar 24 22:24:32 docker03 systemd[1]: etcd.service failed.

Mar 24 22:24:33 docker03 systemd[1]: etcd.service holdoff time over, scheduling restart.

Mar 24 22:24:33 docker03 systemd[1]: start request repeated too quickly for etcd.service

Mar 24 22:24:33 docker03 systemd[1]: Failed to start Etcd Server.

-- Subject: Unit etcd.service has failed

-- Defined-By: systemd

-- Support: http: //lists .freedesktop.org /mailman/listinfo/systemd-devel

--

-- Unit etcd.service has failed.

--

-- The result is failed.

Mar 24 22:24:33 docker03 systemd[1]: Unit etcd.service entered failed state.

Mar 24 22:24:33 docker03 systemd[1]: etcd.service failed.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

[[email protected] ~] # cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target   [Service] Type=notify EnvironmentFile=- /opt/kubernetes/cfg/etcd ExecStart= /opt/kubernetes/bin/etcd \ --name=${ETCD_NAME} \ --data- dir =${ETCD_DATA_DIR} \ --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http: //127 .0.0.1:2379 \ --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-advertise-peer-u rls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --initial-cluster=${ETCD_INITIAL_CLUSTER} \ --initial-cluster-token=${ETCD_INITIAL_CLUSTER} \ --initial-cluster-state=existing \  # 将new这个参数修改成existing，启动正常！ --cert- file = /opt/kubernetes/ssl/server .pem \ --key- file = /opt/kubernetes/ssl/server-key .pem \ --peer-cert- file = /opt/kubernetes/ssl/server .pem \ --peer-key- file = /opt/kubernetes/ssl/server-key .pem \ --trusted-ca- file = /opt/kubernetes/ssl/ca .pem \ --peer-trusted-ca- file = /opt/kubernetes/ssl/ca .pem Restart=on-failure LimitNOFILE=65536   [Install] WantedBy=multi-user.target

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

[[email protected] ~] # cat /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

 

[Service]

Type=notify

EnvironmentFile=- /opt/kubernetes/cfg/etcd

ExecStart= /opt/kubernetes/bin/etcd \

--name=${ETCD_NAME} \

--data- dir =${ETCD_DATA_DIR} \

--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \

--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http: //127 .0.0.1:2379 \

--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \

--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \

--initial-cluster=${ETCD_INITIAL_CLUSTER} \

--initial-cluster-token=${ETCD_INITIAL_CLUSTER} \

--initial-cluster-state=existing \  # 将new这个参数修改成existing，启动正常！

--cert- file = /opt/kubernetes/ssl/server .pem \

--key- file = /opt/kubernetes/ssl/server-key .pem \

--peer-cert- file = /opt/kubernetes/ssl/server .pem \

--peer-key- file = /opt/kubernetes/ssl/server-key .pem \

--trusted-ca- file = /opt/kubernetes/ssl/ca .pem \

--peer-trusted-ca- file = /opt/kubernetes/ssl/ca .pem

Restart=on-failure

LimitNOFILE=65536

 

[Install]

WantedBy=multi-user.target