Cassandra cluster management-delete abnormal nodes
Fault simulation node: 172.20.101.166, the simulation node system is directly damaged, and all data is lost.
Note:
This document is only a part of the system document. For details of the previous document information, please see:
Test preparation + offline normal node: https://blog.51cto. com/michaelkang/2419518
Node restarts abnormally: https://blog.51cto.com/michaelkang/2419524
Add a new node: https://blog.51cto.com/michaelkang/2419521
Remove exception Node: https://blog.51cto.com/michaelkang/2419525
View the logs of other nodes in the cluster
The log print content is basically the same as the abnormal restart of the node.
View abnormal nodes
[[email protected] ~]# nodetool describecluster
Cluster Information:
Name: pttest
Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
DynamicEndPointSnitch: enabled
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
cfce5a85-19c8-327a-ab19- e1faae2358f7: [172.20.101.164, 172.20.101.165, 172.20.101.167, 172.20.101.160, 172.20.101.157]
UNREACHABLE: [172.20.101.166]
Remedy
1: Execute the nodetool status command on a normal node to obtain the id of the down node
[[email protected] ~]# nodetool status
Datacenter: dc1
< p>Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
— Address Load Tokens Owns Host ID Rack
…….
The abnormal node status is DN
DN 172.20.101.166 76.83 MiB 256? 88e16e35-50dd-4ee3-aa1a-f10a8c61a3eb rack1
UN 172.20.101.167 71.78 MiB 256? 8808aaf7-690c-4f0c-be9b-ce655c1464d4 rack1
UN 172.20.101.160 66 .31 MiB 256? 57cc39fc-e47b-4c96-b9b0-b004f2b79242 rack1
UN 172.20.101.157 55 MiB 256? 091ff0dc-415b-48a7-b4ce-e70c84bbfafc rack1
2: Confirm whether the abnormal node is Seed node, if the dead node is a seed node, please change the seed node configuration of the cluster on each node:
In the cassandra.yaml file of each node, from the seed-provider attribute-seeds Delete the IP address of the invalid node from the list. If the cluster needs a new seed node to replace the dead seed node, please add the IP address of the new node to the seeds list. The node service needs to be restarted.
3: If it is an ordinary node, execute: nodetool removenode host-id to remove the node
nodetool removenode 88e16e35-50dd-4ee3-aa1a-f10a8c61a3eb
Task After the execution is successful, the cluster will detect the data copy and restore the necessary copy.
### Verify node deletion
The abnormal node has disappeared, and the node was successfully deleted.
[[email protected] ~]# nodetool status
Datacenter: dc1
Status=Up/Down|/ State=Normal/Leaving/Joining/Moving– Address Load Tokens Owns Host ID RackUN 172.20.101.164 68.2 MiB 256? Dcbbad83-fe7c-4580-ade7-aa763b8d2c40 rack1UN 172.20.101.165 59.06 MiB 256? B985de23-6ad1-40b9-a252-dbaeb5d4cb12 rack1UN 172.20.17-690 8c 4f0c-be9b-ce655c1464d4 rack1UN 172.20.101.160 66.32 MiB 256? 57cc39fc-e47b-4c96-b9b0-b004f2b79242 rack1UN 172.20.101.157 55.06 MiB 256? 091ff0dc-415b-48a7-b4fa-e70
>