标签归档: ha

CentOS 6.2（RHEL6）高可用服务器环境搭建

目标

为四台新服务器部署MySQL、Coreseek、Redis、MongoDB、NFS、Varnish等服务。四台服务器分为两组、使用Linux-HA + Pacemaker 实现高可用（HA, High-Availablility）。

生产服务器基于安全原因考虑不会安装gcc等编译工具，因此所有软件的编译工作都在专门的编译机上完成。编译完成后再部署到生产环境，依赖用yum、rpm解决。

 

硬件环境

1-DELL R610 6核*2，Xeon5690(3.46G) 8G*6 2.5寸SAS10000转300G*2
2-DELL R610 4核*2，Xeon5620(2.13G) 8G*6 2.5寸SAS10000转300G*2
3-DELL R610 4核*2，Xeon5620(2.13G) 8G*6 2.5寸SAS10000转300G*2
4-DELL R610 6核*2，Xeon5690(3.46G) 8G*6 2.5寸SAS10000转300G*2
MD3200 3.5寸SAS15000转300G*12

 

Linux-HA  3.0

Linux-HA  3.0由三部分组成：Heartbeat、Cluster Glue、Resource Agents

  1. hearbeat本身是整个集群的基础（cluster messaging layer），负责维护集群各节点的信息以及它们之前通信；
  2. cluster-glue相当于一个中间层，可以将heartbeat和crm（pacemaker）联系起来，主要包含2个部分，LRM和STONITH；
  3. resource-agent，就是各种的资源的ocf脚本，这些脚本将被LRM调用从而实现各种资源启动、停止、监控等等。
Note
直到 Heartbeat release 2.1.3 , Pacemaker 和 Heartbeat 都是 Linux-HA 保护伞项目的一部分. 在3.0版后, Pacemaker 被作为一个独立项目分割了出去, 且依然完全支持 Heartbeat .

Pacemaker

Pacemaker = CRM = Cluster Resource Manager，

 

通过yum安装

 

yum install cluster-glue resource-agents pacemaker

Installed:
cluster-glue.x86_64 0:1.0.5-2.el6 pacemaker.x86_64 0:1.1.6-3.el6 resource-agents.x86_64 0:3.9.2-7.el6

Dependency Installed:
cifs-utils.x86_64 0:4.8.1-5.el6 cluster-glue-libs.x86_64 0:1.0.5-2.el6 clusterlib.x86_64 0:3.0.12.1-23.el6_2.1
corosync.x86_64 0:1.4.1-4.el6_2.2 corosynclib.x86_64 0:1.4.1-4.el6_2.2 gnutls.x86_64 0:2.8.5-4.el6_2.2
keyutils.x86_64 0:1.4-3.el6 libevent.x86_64 0:1.4.13-1.el6 libgssglue.x86_64 0:0.1-11.el6
libibverbs.x86_64 0:1.1.5-3.el6 libnl.x86_64 0:1.1-14.el6 librdmacm.x86_64 0:1.0.14.1-3.el6
libtalloc.x86_64 0:2.0.1-1.1.el6 libtasn1.x86_64 0:2.3-3.el6_2.1 libtirpc.x86_64 0:0.2.1-5.el6
lm_sensors-libs.x86_64 0:3.1.1-10.el6 net-snmp-libs.x86_64 1:5.5-37.el6_2.1 nfs-utils.x86_64 1:1.2.3-15.el6_2.1
nfs-utils-lib.x86_64 0:1.1.5-4.el6 pacemaker-cli.x86_64 0:1.1.6-3.el6 pacemaker-cluster-libs.x86_64 0:1.1.6-3.el6
pacemaker-libs.x86_64 0:1.1.6-3.el6 perl-TimeDate.noarch 1:1.16-11.1.el6 quota.x86_64 1:3.17-16.el6
rpcbind.x86_64 0:0.2.0-8.el6 tcp_wrappers.x86_64 0:7.6-57.el6

 

环境配置

echo /opt/servers/lib > /etc/ld.so.conf.d/servers.conf
groupadd -g 1000 mysql; useradd -g mysql -u 1000 mysql
groupadd -g 3000 wwwroot; useradd -g wwwroot -u 3000 wwwroot

编译环境

CentOS 6.2 x86_64 Linux 2.6.32-220.el6.x86_64

编译环境要求

  • wget
  • mercurial
  • gcc
  • autoconf
  • automake
  • make
  • libtool
  • pkgconfig
  • glib2-devel
  • libxml2 libxml2-devel
  • bzip2-devel
  • libtool-ltdl-devel
  • libuuid-devel
  • libxslt-devel

 

编译glue

./autogen.sh
./configure --prefix=/opt/servers
make
make install

./configure 的输出

cluster-glue configuration:
  Version                  = 1.0.9 (Build: 0a08a469fdc8a0db1875369497bc83c0523ceb21)
  Features                 =

  Prefix                   = /opt/servers
  Executables              = /opt/servers/sbin
  Man pages                = /opt/servers/share/man
  Libraries                = /opt/servers/lib
  Header files             = /opt/servers/include
  Arch-independent files   = /opt/servers/share
  Documentation            = /opt/servers/share/doc/cluster-glue
  State information        = /opt/servers/var
  System configuration     = /opt/servers/etc

  Use system LTDL          = yes

  HA group name            = haclient
  HA user name             = hacluster

  CFLAGS                   = -g -O2 -ggdb3 -O0  -fgnu89-inline -fstack-protector-all -Wall -Waggregate-return -Wbad-function-cast -Wcast-qual -Wcast-align -Wdeclaration-after-statement -Wendif-labels -Wfloat-equal -Wformat=2 -Wformat-security -Wformat-nonliteral -Winline -Wmissing-prototypes -Wmissing-declarations -Wmissing-format-attribute -Wnested-externs -Wno-long-long -Wno-strict-aliasing -Wpointer-arith -Wstrict-prototypes -Wwrite-strings -ansi -D_GNU_SOURCE -DANSI_ONLY -Werror
  Libraries                = -lbz2 -lxml2 -lc -luuid -lrt -ldl  -lglib-2.0   -lltdl
  Stack Libraries          =

注意 Use system LTDL， 在未安装lib tool-ltdl-devel之前该项会显示为no，且编译会报错

Making all in libltdl
gmake[1]: Entering directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/libltdl'
gmake[1]: *** 没有规则可以创建目标“all”。 停止。
gmake[1]: Leaving directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/libltdl'
make: *** [all-recursive] 错误 1

另外一个可能遇到的错误是关于uuid的，简单的yum install libuuid-devel可以解决，在解决这个问题的时候我google了一下，如果不行可能还会需要在configure的时候带上参数 LIBS=’/lib/libuuid.so.1’（注意，如果是64位则是/lib64/….）。

./.libs/libplumb.so: undefined reference to `uuid_parse'
./.libs/libplumb.so: undefined reference to `uuid_generate'
./.libs/libplumb.so: undefined reference to `uuid_copy'
./.libs/libplumb.so: undefined reference to `uuid_is_null'
./.libs/libplumb.so: undefined reference to `uuid_unparse'
./.libs/libplumb.so: undefined reference to `uuid_clear'
./.libs/libplumb.so: undefined reference to `uuid_compare'
collect2: ld returned 1 exit status
gmake[2]: *** [ipctest] 错误 1
gmake[2]: Leaving directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/lib/clplumbing'
gmake[1]: *** [all-recursive] 错误 1
gmake[1]: Leaving directory `/root/build/Reusable-Cluster-Components-glue--glue-1.0.9/lib'
make: *** [all-recursive] 错误 1

编译Heartbeat 3

./bootstrap
LIBS="/lib64/libuuid.so.1" CFLAGS="-I/opt/servers/include -L/opt/servers/lib" ./configure --prefix=/opt/servers
make
make install

因为我们的安装路径非标准路径，所以这里要指定一下CFLAGS 的include path，否则configure会因找不到glue_config.h而报错

checking for heartbeat/glue_config.h... no
configure: error: in `/root/build/Heartbeat-3-0-7e3a82377fa8':
configure: error: Core development headers were not found
See `config.log' for more details.

虽然已经在ldconfig 里加上了/opt/servers/lib 但是如果不在CFLAGS里指定 lib path 还是会导致 ld 找不到 -lplumb （/opt/servers/lib/libplumb.so）。
另外如果没有安装libxslt-devel在make的时候还会导致 “gmake[1]: –xinclude: Command not found”。

./configure 输出

heartbeat configuration:
  Version                  = "3.0.5"
  Executables              = "/opt/servers/sbin"
  Man pages                = "/opt/servers/share/man"
  Libraries                = "/opt/servers/lib64"
  Header files             = "/opt/servers/include"
  Arch-independent files   = "/opt/servers/share"
  Documentation files      = "/opt/servers/share/doc/heartbeat"
  State information        = "/opt/servers/var"
  System configuration     = "/opt/servers/etc"
  Init (rc) scripts        = "/etc/init.d"
  Init (rc) defaults       = "/etc/sysconfig"
  Use system LTDL          = "yes"
  HA group name            = "haclient"
  HA group id              = "65"
  HA user name             = "hacluster"
  HA user user id          = "17"
  Build dopd plugin        = "yes"
  Enable times kludge      = "yes"
  CC_WARNINGS              = " -Wall -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-qual -Wcast-align -Wbad-function-cast -Winline -Wmissing-format-attribute -Wformat=2 -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Werror "
  Mangled CFLAGS           = "-I/opt/servers/include -L/opt/servers/lib -I/opt/servers/include/heartbeat  -Wall -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wdeclaration-after-statement -Wpointer-arith -Wwrite-strings -Wcast-qual -Wcast-align -Wbad-function-cast -Winline -Wmissing-format-attribute -Wformat=2 -Wformat-security -Wformat-nonliteral -Wno-long-long -Wno-strict-aliasing -Werror  -ggdb3 -funsigned-char"
  Libraries                = "-lbz2 -lz -lc -luuid -lrt -ldl /lib64/libuuid.so.1 -lltdl"
  RPATH enabled            = ""
  Distro-style RPMs        = "no"

Note: If you use the 'make install' method for installation you
also need to adjust '/etc/passwd' and '/etc/group' manually.

到这里 cluster-glue 和 heartbeat 都编译完成了。

然后老大说要用rpm装。。。我了个去。。。

然后centos 6.2的源里 cluster-glue、resource-agents、pacemaker 都有，就是没有heartbeat。 幸好heartbeat的source里提供了.spec 可以用rpmbuild来生成rpm包。

下面继续 heartbeat.rpm 的 build过程

首先把下载下来的heartbeat源码包解压一份，然后再把源码包(.tar.gz)复制到 /root/rpmbuild/SOURCES/heartbeat.tar.bz2

tar -jxf Heartbeat-3.0.5.tar.bz2
cp Heartbeat-3.0.5.tar.bz2 /root/rpmbuild/SOURCES/heartbeat.tar.bz2
cd Heartbeat-XXXXXX
# 解决依赖
yum install ncurses-devel openssl-devel gettext bison flex mailx cluster-glue-libs-devel docbook-dtds docbook-style-xsl
# 开始生成。 （-bb 只编译二进制rpm包 -bs 只编译源码rpm包 -ba 同时编译二进制和源码rpm包）
rpmbuild -ba heartbeat-fedora.spec

最后生成文件

Wrote: /root/rpmbuild/SRPMS/heartbeat-3.0.5-1.el6.src.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/heartbeat-3.0.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/heartbeat-libs-3.0.5-1.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/heartbeat-devel-3.0.5-1.el6.x86_64.rpm

 

未完未完。。。
中途被拉去解决别的问题，HA的部署工作被人接手掉了

Read: 288