Heartbeat is a highly available cluster system based on Linux open source. It mainly includes two highly available cluster components, heartbeat service and resource takeover. The heartbeat monitoring service can be performed through network links and serial ports, and supports redundant links. They send messages to each other to tell each other their current status. If they do not receive the message sent by the other party within a specified time, then It is considered that the other party is invalid, and then the resource takeover module needs to be activated to take over the resources or services running on the other party’s host. This article briefly describes the heartbeat v2 cluster architecture components and related concepts for your reference.

One, the characteristics of high-availability clusters

High-availability services

< pre>Usually use the cluster method to achieve, this is also the biggest role and embodiment of the cluster.
The ultimate goal is to ensure that the service is available in real time, and will not be terminated or unavailable due to any software and hardware failures.

Metrics

Measure the reliability and maintainability of the system. 
In engineering, the mean time to failure (MTTF) is usually used to measure the reliability of the system, and the mean time to repair (MTTR) is used to measure the maintainability of the system. 
 Calculation formula, HA=MTTF/(MTTF+MTTR)*100%
 99% The annual downtime does not exceed 4 days
 99.9% The annual downtime does not exceed 10 hours
 99.99% annual downtime does not exceed 1 hour
 99.999% annual downtime does not exceed 6 minutes

cluster node

cluster software must Include a mechanism to define which systems can be used as cluster nodes (define nodes, 2 nodes or more). 
All hosts in the cluster are called nodes. 

Cluster Services and Resources

Which services or applications can failover between nodes, and interconnection can transmit communication between nodes. 
Services usually include multiple resources, and multiple resources make up a certain service. 
Such as mysql high-availability services, vip, mysqld, shared or mirrored disks, etc. are the resources required by the service. 
The management of cluster services is actually the management of resources. 

Resource isolation and split-brain

Due to software and hardware failures, nodes are down and resource contention occurs, that is, a failed node or a normal coexistence situation . 
In the case that the failed node controls the same cluster resources, implement resource isolation to prevent split brain (Fence mechanism, STONITH, etc.). 

cluster status monitoring

Use cluster management and monitoring tools and predefined scripts to configure common services or applications, monitoring, failover, etc. 
The most well-known one is the heartbeat, which is mainly used to perceive each other's existence in a cluster environment. 
It can be based on serial port, multicast, broadcast and multicast communication mechanisms. Once the heartbeat fails, actions such as corresponding resource transfer and cluster reconstruction will occur. 

Two, HeartBeat components

Heartbeat is a highly available cluster system based on Linux open source. It mainly includes two high-availability cluster components, heartbeat service and resource takeover. The major version changes are mainly divided into three stages.

1, Heartbeat 1.x component

Heartbeat1.x allows cluster nodes and resources to pass through /etc/ha. The two files under the d directory are used to configure
ha.cf: define cluster nodes, failure detection and switching time intervals, cluster time log mechanism and node fence method
haresources:
define cluster resource groups, each line Define a default node and a group of resources that can failover together. Resources include IP address, file system, service or application

2, Heartbeat 2 .x component

Heartbeat 2.0 introduces a module structure configuration method based on Heartbeat1.x configuration, cluster resource manager (Cluster Rescource Manager-CRM).
CRM model can support Up to 16 nodes, this model uses XML-based cluster information (Cluster Information Base-CIB) configuration.
The last official STABLE release 2.x version of Heartbeat 2.x is 2.1.4.
The CIB file (/var/lib/heartbeat/crm/cib.xml) will be automatically copied between each node. It defines the following objects and actions:
*Cluster node
*Cluster resources, including Attributes, priorities, groups and dependencies
*Log, monitoring, arbitration and fence standards
*Actions that need to be performed when the service fails or the standards set in it are met

< p>

messaging and infrastructure layer (Messaging and Infrastructure Layer)

The primary or first layer is the messaging/infrastructure layer, also called For the heartbeat layer. #Author:Leshami
This layer contains components that send heartbeat messages containing the "I'm still alive" signal, and other information. 
The Heartbeat program resides in the message/infrastructure layer. #Blog:http://blog.csdn.net/leshami

Membership Layer

Membership Layer from the bottom is the heartbeat layer Obtain information, be responsible for calculating the maximum full connection setting of the cluster node and synchronizing it to all members on the node. 
This layer is responsible for the consistency between cluster members and provides the cluster topology to the upper-level components. 

Resource Allocation Layer

The third layer is the resource allocation layer. This layer is the most complex and consists of the following parts:
Cluster Resource Manager
Every action in the resource allocation layer is managed by the Cluster Resource Manager. 
 Any component of the resource allocation layer, or any other higher-level components that need to communicate, are managed by the local cluster resource manager. 

 On each node, the cluster resource manager maintains a cluster information base, or CIB (see cluster information base below). 
 A node in the cluster will be selected as the designated coordinator (DC), which means it has a master CIB. All other CIBs in the cluster are copies of the master CIB. 
 Normal CIB read and write operations are serialized through the main CIB. 
 In a cluster, the DC can determine the related changes that need to be performed for a cluster-wide change, such as isolating a node or moving resources. 

Cluster Information Base (Cluster Information Base) 
 The cluster information base or CIB is the entire cluster configuration and status, including node members, resource constraints, etc. It is an XML file that resides in memory. 
 In the cluster, there is a master CIB maintained by the DC, and all other nodes contain a copy of the CIB. 
 If the administrator wants to manage the cluster, he can use the cibadmin command line tool or the heartbeat GUI tool. 
 The heartbeat GUI tool can be used to connect from any machine to the cluster. 
 The cibadmin command must be used on cluster nodes and is not limited to DC nodes. 

Policy Engine (PE) and Transition Engine (TE)
 Whenever the designated coordinator needs to make a cluster-wide change (reconstructing a new CIB), the policy The engine is used to calculate the next state and (resources) of the cluster to implement the list of operations it needs. 
 The command calculated by the strategy engine is then executed by the conversion engine. 
 DC will send relevant information to the cluster resource manager, and then use its own local resource manager (LRM) to perform necessary resource operations. 
 PE and TE must run in pairs on the DC node. 

LRM (Local Resource Manager)
 The local resource manager calls the local resource agent on behalf of CRM. So it can perform start/stop/monitor operations and report the results to CRM. 
 LRM retains all resource-related information on the local node. 

Resource Layer

The fourth and highest layer is the resource layer. The resource layer includes one or more resource agents (RA). 
The resource agent is a program, usually a shell script, that includes starting, stopping and monitoring a certain service (resource). 
The most common resource agent is the LSB initialization script. However, HeartBeat also supports a more flexible and powerful open cluster architecture resource agent API. 
The agent that provides the heartbeat is written into the OCF specification. The resource agent is only called by the local resource manager. 
The third party can define its own agent in the file system and integrate its own software into the cluster. 

3, Heartbeat 3.x component

After the v3 version, the entire heartbeat project has been functionally split and divided into different Sub-projects are developed separately. But the HA implementation principle is basically the same as Heartbeat 2.x, and the configuration is basically the same. After the v3 version, it was split into heartbeat, pacemaker (heart pacemaker), and cluster-glue (cluster gluer). The architecture is separated and can be combined with other components.
The first version of Heartbeat 3 officially released is 3.0.2. The original CRM management was replaced by pacemaker, and the underlying message layer can still use heartbeat v3 or corosync. The specific details will not be introduced in this article, you can refer to clusterlabs.org separately.

Three, heartbeat cluster processing flow

Any behavior performed in the cluster will result in changes to the entire cluster. These operations include things like adding or deleting cluster resources or changing resource limits. When performing such operations, it is important to understand what will happen in the cluster.

For example, suppose you need to add a cluster IP address resource. To do this, use the cibadmin command line tool or the Heartbeat GUI tool to modify the main CIB. It does not require the use of cibadmin commands or GUI tools on the designated coordinator. You can use any tool on any node in the cluster, and the local CIB will replay the requested changes to the designated coordinator. Then specify that the coordination will replicate the CIB changes to all cluster nodes and start the conversion process.

With the help of the strategy engine and the transition engine, the designated coordinator obtains a series of steps that need to be completed in the cluster, possibly on multiple nodes. The designated coordinator sends commands to other cluster resource managers through the message layer.

If necessary, other cluster resource managers use their local resource managers to perform resource modifications and return the results to the designated coordinator. Once the TE on the designated coordination concludes that all necessary operations in the cluster have been successfully completed, the cluster will return to the idle state and wait for further events.

If any operation does not proceed as planned, the strategy engine again calls the new information recorded in the CIB.

When a service or node dies, the same thing will happen. The designated coordinator will be notified by the cluster consistent member service (death on a node) or local resource management (in case of a failed monitor operation). The designated coordinator needs to determine the behavior that will change to a new cluster state. The new cluster state will be represented by a new CIB.