目标
为四台新服务器部署MySQL、Coreseek、Redis、MongoDB、NFS、Varnish等服务。四台服务器分为两组、使用Linux-HA + Pacemaker 实现高可用（HA, High-Availablility）。
生产服务器基于安全原因考虑不会安装gcc等编译工具，因此所有软件的编译工作都在专门的编译机上完成。编译完成后再部署到生产环境，依赖用yum、rpm解决。
硬件环境
| 1-DELL R610 |
6核*2，Xeon5690(3.46G) |
8G*6 |
2.5寸SAS10000转300G*2 |
| 2-DELL R610 |
4核*2，Xeon5620(2.13G) |
8G*6 |
2.5寸SAS10000转300G*2 |
| 3-DELL R610 |
4核*2，Xeon5620(2.13G) |
8G*6 |
2.5寸SAS10000转300G*2 |
| 4-DELL R610 |
6核*2，Xeon5690(3.46G) |
8G*6 |
2.5寸SAS10000转300G*2 |
| MD3200 |
|
|
3.5寸SAS15000转300G*12 |
Linux-HA 3.0
Linux-HA 3.0由三部分组成：Heartbeat、Cluster Glue、Resource Agents
- hearbeat本身是整个集群的基础（cluster messaging layer），负责维护集群各节点的信息以及它们之前通信；
- cluster-glue相当于一个中间层，可以将heartbeat和crm（pacemaker）联系起来，主要包含2个部分，LRM和STONITH；
- resource-agent，就是各种的资源的ocf脚本，这些脚本将被LRM调用从而实现各种资源启动、停止、监控等等。
| Note |
| 直到 Heartbeat release 2.1.3 , Pacemaker 和 Heartbeat 都是 Linux-HA 保护伞项目的一部分. 在3.0版后, Pacemaker 被作为一个独立项目分割了出去, 且依然完全支持 Heartbeat . |
Pacemaker
Pacemaker = CRM = Cluster Resource Manager，
通过yum安装
yum install cluster-glue resource-agents pacemaker
Installed:
cluster-glue.x86_64 0:1.0.5-2.el6 pacemaker.x86_64 0:1.1.6-3.el6 resource-agents.x86_64 0:3.9.2-7.el6
Dependency Installed:
cifs-utils.x86_64 0:4.8.1-5.el6 cluster-glue-libs.x86_64 0:1.0.5-2.el6 clusterlib.x86_64 0:3.0.12.1-23.el6_2.1
corosync.x86_64 0:1.4.1-4.el6_2.2 corosynclib.x86_64 0:1.4.1-4.el6_2.2 gnutls.x86_64 0:2.8.5-4.el6_2.2
keyutils.x86_64 0:1.4-3.el6 libevent.x86_64 0:1.4.13-1.el6 libgssglue.x86_64 0:0.1-11.el6
libibverbs.x86_64 0:1.1.5-3.el6 libnl.x86_64 0:1.1-14.el6 librdmacm.x86_64 0:1.0.14.1-3.el6
libtalloc.x86_64 0:2.0.1-1.1.el6 libtasn1.x86_64 0:2.3-3.el6_2.1 libtirpc.x86_64 0:0.2.1-5.el6
lm_sensors-libs.x86_64 0:3.1.1-10.el6 net-snmp-libs.x86_64 1:5.5-37.el6_2.1 nfs-utils.x86_64 1:1.2.3-15.el6_2.1
nfs-utils-lib.x86_64 0:1.1.5-4.el6 pacemaker-cli.x86_64 0:1.1.6-3.el6 pacemaker-cluster-libs.x86_64 0:1.1.6-3.el6
pacemaker-libs.x86_64 0:1.1.6-3.el6 perl-TimeDate.noarch 1:1.16-11.1.el6 quota.x86_64 1:3.17-16.el6
rpcbind.x86_64 0:0.2.0-8.el6 tcp_wrappers.x86_64 0:7.6-57.el6
环境配置
echo /opt/servers/lib > /etc/ld.so.conf.d/servers.conf
groupadd -g 1000 mysql; useradd -g mysql -u 1000 mysql
groupadd -g 3000 wwwroot; useradd -g wwwroot -u 3000 wwwroot
编译环境
CentOS 6.2 x86_64 Linux 2.6.32-220.el6.x86_64
编译环境要求
- wget
- mercurial
- gcc
- autoconf
- automake
- make
- libtool
- pkgconfig
- glib2-devel
- libxml2 libxml2-devel
- bzip2-devel
- libtool-ltdl-devel
- libuuid-devel
- libxslt-devel
编译glue
./autogen.sh
./configure --prefix=/opt/servers
make
make install
./configure 的输出
cluster-glue configuration:
Version = 1.0.9 (Build: 0a08a469fdc8a0db1875369497bc83c0523ceb21)
Features =
Prefix = /opt/servers
Executables = /opt/servers/sbin
Man pages = /opt/servers/share/man
Libraries = /opt/servers/lib
Header files = /opt/servers/include
Arch-independent files = /opt/servers/share
Documentation = /opt/servers/share/doc/cluster-glue
State information = /opt/servers/var
System configuration = /opt/servers/etc
Use system LTDL = yes
HA group name = haclient
HA user name = hacluster
CFLAGS = -g -O2 -ggdb3 -O0 -fgnu89-inline -fstack-protector-all -Wall -Waggregate-return -Wbad-function-cast -Wcast-qual -Wcast-align -Wdeclaration-after-statement -Wendif-labels -Wfloat-equal -Wformat=2 -Wformat-security -Wformat-nonliteral -Winline -Wmissing-prototypes -Wmissing-declarations -Wmissing-format-attribute -Wnested-externs -Wno-long-long -Wno-strict-aliasing -Wpointer-arith -Wstrict-prototypes -Wwrite-strings -ansi -D_GNU_SOURCE -DANSI_ONLY -Werror
Libraries = -lbz2 -lxml2 -lc -luuid -lrt -ldl -lglib-2.0 -lltdl
Stack Libraries =
注意 Use system LTDL， 在未安装lib tool-ltdl-devel之前该项会显示为no，且编译会报错
Making all in libltdl
gmake[1]: Entering directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/libltdl'
gmake[1]: *** 没有规则可以创建目标“all”。 停止。
gmake[1]: Leaving directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/libltdl'
make: *** [all-recursive] 错误 1
另外一个可能遇到的错误是关于uuid的，简单的yum install libuuid-devel可以解决，在解决这个问题的时候我google了一下，如果不行可能还会需要在configure的时候带上参数 LIBS=’/lib/libuuid.so.1’（注意，如果是64位则是/lib64/….）。
./.libs/libplumb.so: undefined reference to `uuid_parse'
./.libs/libplumb.so: undefined reference to `uuid_generate'
./.libs/libplumb.so: undefined reference to `uuid_copy'
./.libs/libplumb.so: undefined reference to `uuid_is_null'
./.libs/libplumb.so: undefined reference to `uuid_unparse'
./.libs/libplumb.so: undefined reference to `uuid_clear'
./.libs/libplumb.so: undefined reference to `uuid_compare'
collect2: ld returned 1 exit status
gmake[2]: *** [ipctest] 错误 1
gmake[2]: Leaving directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/lib/clplumbing'
gmake[1]: *** [all-recursive] 错误 1
gmake[1]: Leaving directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/lib'
make: *** [all-recursive] 错误 1
编译Heartbeat 3
./bootstrap
LIBS="/lib64/libuuid.so.1" CFLAGS="-I/opt/servers/include -L/opt/servers/lib" ./configure --prefix=/opt/servers
make
make install
因为我们的安装路径非标准路径，所以这里要指定一下CFLAGS 的include path，否则configure会因找不到glue_config.h而报错
checking for heartbeat/glue_config.h... no
configure: error: in `/root/build/Heartbeat-3-0-7e3a82377fa8':
configure: error: Core development headers were not found
See `config.log' for more details.
虽然已经在ldconfig 里加上了/opt/servers/lib 但是如果不在CFLAGS里指定 lib path 还是会导致 ld 找不到 -lplumb （/opt/servers/lib/libplumb.so）。
另外如果没有安装libxslt-devel在make的时候还会导致 “gmake[1]: –xinclude: Command not found”。
./configure 输出
heartbeat configuration:
Version = "3.0.5"
Executables = "/opt/servers/sbin"
Man pages = "/opt/servers/share/man"
Libraries = "/opt/servers/lib64"
Header files = "/opt/servers/include"
Arch-independent files = "/opt/servers/share"
Documentation files = "/opt/servers/share/doc/heartbeat"
State information = "/opt/servers/var"
System configuration = "/opt/servers/etc"
Init (rc) scripts = "/etc/init.d"
Init (rc) defaults = "/etc/sysconfig"
Use system LTDL = "yes"
HA group name = "haclient"
HA group id = "65"
HA user name = "hacluster"
HA user user id = "17"
Build dopd plugin = "yes"
Enable times kludge = "yes"
CC_WARNINGS = " -Wall -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-qual -Wcast-align -Wbad-function-cast -Winline -Wmissing-format-attribute -Wformat=2 -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Werror "
Mangled CFLAGS = "-I/opt/servers/include -L/opt/servers/lib -I/opt/servers/include/heartbeat -Wall -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-qual -Wcast-align -Wbad-function-cast -Winline -Wmissing-format-attribute -Wformat=2 -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Werror -ggdb3 -funsigned-char"
Libraries = "-lbz2 -lz -lc -luuid -lrt -ldl /lib64/libuuid.so.1 -lltdl"
RPATH enabled = ""
Distro-style RPMs = "no"
Note: If you use the 'make install' method for installation you
also need to adjust '/etc/passwd' and '/etc/group' manually.
到这里 cluster-glue 和 heartbeat 都编译完成了。
然后老大说要用rpm装。。。我了个去。。。
然后centos 6.2的源里 cluster-glue、resource-agents、pacemaker 都有，就是没有heartbeat。 幸好heartbeat的source里提供了.spec 可以用rpmbuild来生成rpm包。
下面继续 heartbeat.rpm 的 build过程
首先把下载下来的heartbeat源码包解压一份，然后再把源码包(.tar.gz)复制到 /root/rpmbuild/SOURCES/heartbeat.tar.bz2
tar -jxf Heartbeat-3.0.5.tar.bz2
cp Heartbeat-3.0.5.tar.bz2 /root/rpmbuild/SOURCES/heartbeat.tar.bz2
cd Heartbeat-XXXXXX
# 解决依赖
yum install ncurses-devel openssl-devel gettext bison flex mailx cluster-glue-libs-devel docbook-dtds docbook-style-xsl
# 开始生成。 （-bb 只编译二进制rpm包 -bs 只编译源码rpm包 -ba 同时编译二进制和源码rpm包）
rpmbuild -ba heartbeat-fedora.spec
最后生成文件
Wrote: /root/rpmbuild/SRPMS/heartbeat-3.0.5-1.el6.src.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/heartbeat-3.0.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/heartbeat-libs-3.0.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/heartbeat-devel-3.0.5-1.el6.x86_64.rpm
未完未完。。。
中途被拉去解决别的问题，HA的部署工作被人接手掉了
