Heartbeat3.xThe whole application strategy:TestHeartbeat的HAFunction
1. Start Heartbeat
1, start the Heartbeat of the master node h2>
HeartbeatAfter the installation is complete, it will automatically enter /etc/init.d< The startup script file heartbeat is generated in the /span> directory, directly enter /etc/init.d/heartbeat to see the usage of the heartbeat script, as shown below:
|
[root@node1 ~]# /etc/ init.d/heartbeat Usage:/etc/init.d/heartbeat {start|stop|status|restart|reload|force-reload} < p align="left"> |
< p align="left">So starting heartbeat can be done with the following command:
|
[root@node1~]#service heartbeat start or through [root@ node1~]#/etc/init.d/heartbeat start
|
This starts the heartbeat service of the master node
The log information is as follows:
< table border="1" cellspacing="0" cellpadding="0">
Feb 5 19:09: 48 node1 heartbeat: [22768]: info: glib: ucast: bound send socket to device: eth0
Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: bound receive socket to device: eth0
Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ucast: started on port 694 interface eth0 to 192.168.12.1
Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ping heartbeat started.
< p style="background:#D9D9D9">Feb 5 19:09:48 node1 heartbeat: [22768]: info: glib: ping group heartbeat started.
Feb 5 19:09:48 node1 heartbeat: [22768]: info: Local status now set to:’up’
Feb 5 19:09:49 node1 heartbeat: [22768 ]: info: Link 192.168.12.1:192.168.12.1 up.
Feb 5 19:09:49 node1hear tbeat: [22768]: info: Status update for node 192.168.12.1: status ping
Feb 5 19:09:49 node1 heartbeat: [22768]: info: Link group1:group1 up.
Feb 5 19:09:49 node1 heartbeat: [22768]: info: Status update for node group1: status ping
< span style="color:#555555"> This log is Heartbeat in the initial configuration, for example, the heartbeat interval of heartbeat, UDP broadcast port , The running status of the ping node, etc. The log information will be suspended here. After 120 seconds, heartbeat will continue to output the log, and this 120 seconds is just the time set by the “initdead” option in ha.cf. The output information of heartbeat at this time is as follows:
|
Feb 5 19:11:48 node1 heartbeat: [22768]: WARN: node node2: is dead Feb 5 19:11 :48 node1 heartbeat: [22768]: info: Comm_now_up(): updating status to active Feb 5 19:11:48 node1 heartbeat: [22768]: info: Local status now set to:’active’ Feb 5 19:11:48 node1 heartbeat: [22768]: info: Starting child client”/usr/local/ha /lib/heartbeat/pingd -m 100 -d 5s” (102,105) Feb 5 19:11:49 node1 heartbeat: [22768]: WARN: No STONITH device configured. Feb 5 19:11:49 node1 heartbeat: [22768]: WARN: Shared disks are not protected. Feb 5 1 9:11:49 node1 heartbeat: [22768]: info: Resources being acquired from node2. Feb 5 19:11:49 node1 heartbeat: [22794]: info : Starting”/usr/local/ha/lib/heartbeat/pingd -m 100 -d 5s” as uid 102 gid 105 (pid 22794)
|
In the above log, because node2 has not started , So it will give a warning message of “node2: is dead”, and then start the heartbeat plug-in pingd. Since we did not configure STONITH in the ha.cf file, the log also gave a warning prompt of “No STONITH deviceconfigured” .
Continue to look at the following log:
|
Feb 5 19:11:50 node1 IPaddr[22966]: INFO: Resource is stopped Feb 5 19:11:50 node1 ResourceManager[22938]: info: Running /usr/local/ha/etc/ha.d/resource.d/ IPaddr 192.168.12.135 start Feb 5 19:11:50 node1 IPaddr[23029]: INFO: Using calculated nic for 192.168.12.135: eth0 Feb 5 19:11:50 node1 IPaddr[23029]: INFO: Using calculated netmask for 192.168.12.135: 255.255.255.0 Feb 5 19:11:51 node1 pingd: [22794]: info: attrd_lazy_update: Connecting to cluster… 5 retries remaining Feb 5 19:11:51 node1 IPaddr[23029]: INFO: eval ifconfig eth0:0 192.168.12.135 netmask 255.255.255.0 broadcast 192.168.12.255 Feb 5 19:11:51 node1 avahi-daemon[2455]: Registering new address record for 192.168.12.135 on eth0. Feb 5 19:11:51 node1 IPaddr[23015]: INFO: Success Feb 5 19:11:51 node1 Filesystem[23134]: INFO: Resource is stopped Feb 5 19:11:51 node1 ResourceManager[22938] : Info: Running /usr/local/ha/etc/ha.d/resource.d/Filesystem /dev/sdf1 /data1 ext3 start Feb 5 19:11 :52 node1 Filesystem[23213]: INFO: Running start for /dev/sdf1 on/data1 Feb 5 19:11:52 node1 kernel: kjournald starting. Commit interval 5 seconds Feb 5 19:11:52 node1 kernel: EXT3 FS on sdf1, internal journal Feb 5 19:11:52 node1 kernel: EXT3-fs: mounted file system with ordered data mode. Feb 5 19:11:52 node1 Filesystem[23205]: INFO: Success < span style="color:#555555"> |
The above log is To monitor and take over resources, mainly complete the settings in the haresources file. Here is to enable the cluster virtual IP and mount the disk partition. At this time, check the network configuration of the master node through the ifconfig command, you can see that the master node The IP address of the cluster has been automatically bound, and the cluster IP address 192.168.12.135 is detected by the ping command on the host outside the HA cluster, and it is already in a communicable state, that is, the address becomes available.
At the same time, check the mounting status of the disk partition and share the disk partition /dev/sdf1 has been automatically mounted.
|
[root@data-1-1 ha.d]# ip add 1: lo: < LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: link/ether 00:0c:29 :f1:05:57 brd ff:ff:ff:ff:ff:ff inet 192.168.1.55/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/ 24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 3: eth1: link/ether 00:0c:29:f1:05:61 brd ff:ff:ff:ff:ff:ff inet 10.1.1.1/24 brd 10.1.1.255 scope global eth1 [root@data-1-1 ha.d] # ip add|grep 192.168.1 inet 192.168.1.55/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0: 0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-1 ha.d]#
|
|
[root@data-1-1 ha.d]# [root@data-1-1 ha.d]# cat /var/log/ha-log Jun 04 16:37:48 data-1-1 heartbeat: [3870]: info: ************************** Jun 04 16:39:49 data-1-1 heartbeat: [ 3871]: info: Resources being acquired from data-1-2. harc[3883]: 2015/06/04_16:39:49 info: Running /usr/local/ha/etc/ha. d//rc.d/status status mach_down[3923]: 2015/06/04_16:39:50 info: Taking over resource group IPaddr::192.168.1.58/24/eth0 IPaddr[3940]: 2015/06/04_16:39:50 INFO: Resource is sto pped Jun 04 16:39:50 data-1-1 heartbeat: [3884]: info: Local Resource acquisition completed. ResourceManager[3993]: 2015/06/04_16 :39:50 info: Acquiring resource group: data-1-2 IPaddr::192.168.1.58/24/eth0 IPaddr[4034]: 2015/06/04_16:39:50 INFO: Resource is stopped ResourceManager[3993]: 2015/06/04_16:39:50 info: Running /usr/local/ha/etc/ha.d/resource.d/IPaddr 192.168.1.58/24/eth0 start IPaddr[4119]: 2015/06/04_16:39:50 INFO: Using calculated netmask for 192.168.1.58: 255.255.255.0 IPaddr[4119]: 2015/06 /04_16:39:50 INFO: eval ifconfig eth0:0 192.168.1.58 netmask 255.255.255.0 broadcast 192.168.1.255 IPaddr[4093]: 2015/06/04_16:39:50 INFO: Success p> mach_down[3923]: 2015/06/04_16:39:50 info: /usr/local/ha/share/heartbeat/mach_down: nice_failback: foreign resources acquired mach_down[3923] : 2015/06/04_16:39:50 info: mach_down takeover complete for node data-1-2. Jun 04 16:39:50 data-1-1 heartbeat: [3 871]: info: mach_down takeover complete. Jun 04 16:39:50 data-1-1 heartbeat: [3871]: info: Initial resource acquisition complete (mach_down) harc[4222]: 2015/06/04_16:39:50 info: Running /usr/local/ha/etc/ha.d//rc.d/ip-request-resp ip-request-resp < p>ip-request-resp[4222]: 2015/06/04_16:39:51 received ip-request-resp IPaddr::192.168.1.57/24/eth0 OK yes ResourceManager[4245]: 2015/06/04_16:39:51 info: Acquiring resource group: data-1-1 IPaddr::192.168.1.57/24/eth0 IPaddr[4273]: 2015/06/04_16:39: 51 INFO: Resource is stopped ResourceManager[4245]: 2015/06/04_16:39:51 info: Running /usr/local/ha/etc/ha.d/resource.d/IPaddr 192.168. 1.57/24/eth0 start IPaddr[4358]: 2015/06/04_16:39:51 INFO: Using calculated netmask for 192.168.1.57: 255.255.255.0 IPaddr[4358 ]: 2015/06/04_16:39:51 INFO: eval ifconfig eth0:1 192.168.1.57 netmask 255.255.255.0 broadcast 192.168.1.255 IPaddr[4332]: 2015/06/04_16:39:51 INFO: Su ccess Jun 04 16:40:00 data-1-1 heartbeat: [3871]: info: Local Resource acquisition completed. (none) Jun 04 16:40:00 data-1-1 heartbeat: [3871]: info: local resource transition completed. [root@data-1-1 ha.d]# |
3. Line brain
1) Node data-1-1:
|
[root@data-1-1 ha.d]# [root@data-1-1 ha.d]# ip add|grep 192.168. 1 inet 192.168.1.55/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 Inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-1 ha.d]# td> |
2) Node data-1-2:
|
[root@data-1-2 ~]# [root@data-1-2 ~]# ip add|grep 192.168.1 inet 192.168.1.56/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data-1-2 ~]# |
< p>
3) Analysis and processing
Through “pingdata-1-1” and “ping data -1-2”, 2 nodes can ping Pass;
Turn off the firewall of node data-1-1 and node data-1-2;
|
[root@data-1-2 ~]# /etc/init .d/iptables status Firewall is stopped. [root@data-1-1 ha.d]# /etc/init.d /iptables stop [root@data-1-1 ha.d]# |
4) Heartbeat restart
Both nodes are shut down;
|
[root@data-1-2 ~]# [ root@data-1-2 ~]# /etc/init.d/heartbeat stop Stopping High-Availability services: [OK] [root@data-1-2 ~]# /etc/init.d/heartbeat start Starting High-Availability services: IPaddr[4368]: INFO: Resource is stopped IPaddr[4414]: INFO: Resource is stopped [root@data]# /td> |
2, start the Heartbeat of the standby node
start The Heartbeat of the backup node is the same as the main node method, use the following command:
|
[root@node2] ~]#/etc/init.d/heartbeat start Or execute [root@node2~] #service heartbeat start
|
This starts the heartbeat service of the standby node. The heartbeat log output information of the standby node corresponds to the primary node. You can see the following through “tail -f /var/log/messages” Output:
|
Feb 19 02:52:15 node2 heartbeat: [26880]: info: Pacemaker support: false Feb 19 02:52:15 node2 heartbeat: [26880]: info: ************************** Feb 19 02 :52:15 node2 heartbeat: [26880]: info: Configuration validated. Starting h eartbeat 3.0.4 Feb 19 02:52:15 node2 heartbeat: [26881]: info: heartbeat: version 3.0.4 Feb 19 02:52:15 node2 heartbeat: [26881]: info: Heartbeat generation: 1297766398 Feb 19 02:52:15 node2 heartbeat : [26881]: info: glib: UDP multicast heartbeat started for group 225.0.0.1 port 694 interface eth0 (ttl=1 loop=0) Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ucast: write socket priority set to IPTOS_LOWDELAY on eth0 Feb 19 02:52: 15 node2 heartbeat: [26881]: info: glib: ucast: bound send socket to device: eth0 Feb 19 02:52:15 node2 heartbeat: [26881]: info: glib: ping heartbeat started. Feb 19 02:52:15 node2 heartbe at: [26881]: info: glib: ping group heartbeat started. Feb 19 02:52:15 node2 heartbeat: [26881]: info: Local status now set to:’up’ Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link node1:eth0 up. Feb 19 02:52:16 node2 heartbeat: [26881]: info: Status update for node node1: status active Feb19 02: 52:16 node2 heartbeat: [26881]: info: Link 192.168.12.1:192.168.12.1 up. Feb 19 02:52:16 node2 heartbeat: [26881] : Info: Status update for node 192.168.12.1: status ping Feb 19 02:52:16 node2 heartbeat: [26881]: info: Link group1:group1 up. Feb 19 02:52:16 node2 harc[26894]: info: Running /usr/local/ha/etc/ha.d//rc.d/status status Feb 19 02:52:17 node2 heartbeat: [26881]: info: Comm_now_up(): updating status to ac tive Feb 19 02:52:17 node2 heartbeat: [26881]: info: Local status now set to:’active’
|
3 Heartbeat implementation of web service high availability practical case 1
3.1 Installation/Configuration/Start httpd
1)data-1-1:
|
[root@data-1-1 httpd]# cd /var/www/ html/ [root@data-1-1 html]# ll total 0 [root@data- 1-1 html]# echo 10.1.1.1> index.html [root@data-1-1 html]# |
|
[root@data-1-2 httpd]# cd /var/www/html/< /p> [root@data-1-2 html]# e cho 10.1.1.2> index.html [root@data-1-2 html]# |
|
# localhost name resolution is handled within DNS itself. # 127.0.0.1 localhost # ::1 localhost 192.168.1.57 www.pmel.com |
3.3 test
1) data- 1-1 node heartbeat stop:
|
[root@data-1-2 html]# ip add|grep 192.168.1 inet 192.168.1.56/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 inet 192.168.1.57/24 brd 192.168.1.255 scope global secondary eth0:1 [root@data- 1-2 html]# [root@data-1-2 html]# |
|
Picture 01 |
2) Start heartbeat of data-1-1 node:
|
[root@data-1-1 share]# service heartbeat start Starting High-Availability services: IPaddr[13474]: INFO: Resource is stopped IPaddr[13520]: INFO: Resource is stopped Resource is stopped OK ] [root@data-1-1 share]# |
|
[root@data-1-2 html]# ip add|grep 192.168.1 inet 192.168.1.56/24 brd 192.168.1.255 scope global eth0 inet 192.168.1.58/24 brd 192.168.1.255 scope global secondary eth0:0 [root@data-1- 2 html]# |
|
|
4 Heartbeat implementation of web service high availability case 2
4.1 Stop httpd in node data-1-1 and node data-1-2
|
[root@data-1-1 share]# [root@data-1-1 share]# service httpd stop Stopping httpd: Stopping httpd: Stopping httpd: [OK ] [root@data-1-1 share]# #It doesn’t need to start automatically even after booting [root@data-1-1 share]# chkconfig httpd off [root@data-1-1 share]# |
|
[root@data-1-2 html]# < /p> [root@data-1-2 html]# service httpd stop Stopping httpd: [OK] [root@data-1-2 html]# #The boot does not need to start automatically [root@data-1-2 h tml]# chkconfig httpd off [root@data-1-2 html]# |
4.2 Backup and modify haresources
1 ) Backup
|
p> [root@data-1-1 ha.d]# cd /usr/local/ha/etc/ha.d/ [root @data-1-1 ha.d]# cp haresources haresources.01 |
2) Modify
| < p>
[root@data-1-1 ha.d]# vi haresources #node-name resource1 resource2 … resourceN #just .linux-ha.org 135.9.216.110 #just.linux-ha.org 135.9.216.110 http #just.linux-ha.org 135.9.216.110 135.9.215.111 135.9 .216.112 httpd #just.linux-ha.org 135.9.216.3/28/eth0/135.9.216.12 httpd #node1 10.0.0.170 Filesystem::/dev/sda1: :/data1::ext2 data-1-1 IPaddr::192.168.1.57/24/eth0 httpd data-1-2 IPaddr::192.168.1.58/24/eth0 |
3) View httpd service
|
[root@data- 1-1 ha.d]# lsof -i :80 [root@data-1-1 ha.d]# |
4.3 Restart the heartbeat service
1) All nodes are restarted
|
[root@data-1-1 ha.d]# service heartbeat start Starting High-Availability services: IPaddr[14745]: INFO: Resource is stopped IPaddr[14791]: INFO: Resource is stopped |
2) View
Node 1:
|
[root@data -1-1 ha.d]# [root@data-1-1 ha.d]# lsof -i :80 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME httpd 15225 root 10u IPv4 24545 0t0 TCP *:http (LISTEN)< /p> httpd 15227 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15228 apache 10u IPv4 24545 apache 0t0 TCP *:http (LISTEN) httpd 15229 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15230 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15231 apache 0t 10u * IPv4 24545 http (LISTEN) httpd 15232 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15233 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) httpd 15234 apache 10u IPv4 24545 0t0 TCP *:http (LISTEN) [root@data-1-1 ha.d]# td> |
Node 2
|
[root@data-1-2 ha.d]# [root@data-1-2 ha.d]# lsof -i :80 [root@data-1-2 ha.d]# |
p>
4.4 Switch server test
1) Stop the heartbeat service in node data-1-1
[root@data-1-1 ha.d]#
[root@data-1-1 ha.d]# service heartbeatstop
Stopping High-Availability services: Stopping High-Availability services: [OK ]
[root@data-1-1 ha.d]# lsof -i :80
[root@data-1-1 ha.d]#
2) Check httpd service status
[root@data-1-2
span>ha.d]# lsof -i :80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
httpd 14912 root 10u IPv4 21811 0t0 TCP *:http (LISTEN )
httpd 14929 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14930 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14931 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14932 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14933 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14934 apache 10u IPv4 21811 (LISTEN) 0t0 LISTEN *: httpd )
httpd 14935 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
httpd 14936 apache 10u IPv4 21811 0t0 TCP *:http (LISTEN)
[root@data-1-2 ha.d]#
p>
2. Test the high availability function of heartbeat
How can I know if the HA cluster is working properly? A simulated environment test is a good method. Before putting the Heartbeat high-availability cluster in the production environment , You need to test the following steps to determine whether HA is working properly:
(1) Normally shut down and restart the heartbeat of the master node
首先在主节点node1上执行“ service heartbeat stop”正常关闭主节点的Heartbeat进程，此时通过ifconfig命令查看主节点网卡信息，正常情况下，应该可以看到主节点已经释放了集群的服务IP地址，同时也释放了挂载的共享磁盘分区，然后查看备份节点，现在备份节点已经接管了集群的服务IP，同时也自动挂载上了共享的磁盘分区。
在这个过程中，使用ping命令对集群服务IP进行测试，可以看到，集群IP一致处于可通状态，并没有任何延时和阻塞现象，也就是说在正常关闭主节点的情况下，主备节点的切换是无缝的，HA对外提供的服务也可以不间断运行。
接着，将主节点heartbeat正常启动，heartbeat启动后，备份节点将自动释放集群服务IP，同时卸载共享磁盘分区，而主节点将再次接管集群服务IP和挂载共享磁盘分区，其实备份节点释放资源与主节点绑定资源是同步进行的。因而，这个过程也是一个无缝切换。
（2）在主节点上拔去网线
拔去主节点连接公共网络的网线后，heartbeat插件ipfail通过ping测试可以立刻检测到网络连接失败，接着自动释放资源，而就在此时，备用节点的ipfail插件也会检测到主节点出现网络故障，在等待主节点释放资源完毕后，备用节点马上接管了集群资源，从而保证了网络服务不间断持续运行。
同理，当主节点网络恢复正常时，由于设置了“auto_failbackon”选项，集群资源将自动从备用节点切会主节点。
（3）关闭主节点的系统
在主节点拔去电源后，备用节点的heartbeat进程会立刻收到主节点已经shutdown的消息，备用节点就开始进行资源的接管，这种情况其实和主节点网络故障的现象类似。
（4）让主节点系统内核崩溃
当主节点系统崩溃后，网络也就失去了响应，那么备用节点的heartbeat进程就会立刻检测到主节点网络故障，然后进行资源切换，但是由于主节点系统内核崩溃，导致自身不能卸载所占有的资源，例如共享磁盘分区、集群服务IP等，那么此时如果没有类似Stonith设备的话，就会出现资源争用的情况，但是如果有Stonith设备，Stonith设备会首先将故障的主节点电源关闭或者重启此节点等操作，这样就让主节点释放了集群资源，当Stonith设备完成所有操作时，备份节点才拿到接管主节点资源的所有权，从而接管主节点的资源。
（完）
三、错误收集
1）
[root@data-1-1 ha.d]# /etc/init.d/heartbeat
Usage: /etc/init.d/heartbeat{start|stop|status|restart|reload|force-reload}
[root@data-1-1 ha.d]#
[root@data-1-1 ha.d]#
[root@data-1-1 ha.d]# service heartbeatstart
Starting High-Availability services:IPaddr[3096]: INFO: Resource is stopped
IPaddr[3142]: INFO: Resource is stopped
[FAILED]
heartbeat[3200]: 2015/06/04_16:09:04 info:Pacemaker support: no
heartbeat[3200]: 2015/06/04_16:09:04 ERROR:Bad permissions on keyfile [/usr/local/ha/etc/ha.d//authkeys], 600 recommended.
heartbeat[3200]: 2015/06/04_16:09:04 ERROR:Authentication configuration error.
heartbeat[3200]: 2015/06/04_16:09:04 ERROR:Configurat ion error, heartbeat not started.
[root@data-1-1 ha.d]#
处理方法：
|
[root@data-1-1 ha.d]# [root@data-1-1 ha.d]# chmod 600 /usr/local/ha/etc/ha.d/authkeys |