本次部署是基于MFS文件系统环境已经搭建好，具体可参考博文：https://blog.csdn.net/hetoto/article/details/92200785 1.实验环境
rhel 7.3 firewalld is disabled
主机名IP服务server1172.25.12.1mfs master corosync+pacemakerserver2172.25.12.2chunk serverserver3172.25.12.1chunk serverserver4172.25.12.4mfs master corosync+pacemaker物理机172.25.12.250client由于MFS自身的Mfslogger本身的的不可靠性,这里的高可用是对于单节点mfs master来进行配置的,mfs master 处理用户的挂载请求,并且分发数据,很容易成为节点的故障所在
1、配置master，先获取高可用yum源,下载所需软件
[root@server1 ~]# vim /etc/yum.repos.d/rhel7.repo [rhel7.3] name=rhel7.3 baseurl=http://172.25.12.250/rhel7.3 gpgcheck=0 [HighAvailability] name=HighAvailability baseurl=http://172.25.12.250/rhel7.3/addons/HighAvailability gpgcheck=0 [ResilientStorage] name=ResilientStorage baseurl=http://172.25.12.250/rhel7.3/addons/ResilientStorage gpgcheck=0 [root@server1 ~]# yum clean all [root@server1 ~]# yum repolist下载pacemaker、corosync、pcs服务：
[root@server1 ~]# yum install pacemaker corosync pcs -y [root@server1 ~]# rpm -q pacemaker pacemaker-1.1.15-11.el7.x86_642、配置免密登陆
[root@server1 ~]# ssh-keygen [root@server1 ~]# ssh-copy-id server43、启动pcsd服务,设置hacluster用户密码
[root@server1 ~]# systemctl start pcsd [root@server1 ~]# systemctl enable pcsd Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service. [root@server1 ~]# passwd hacluster Changing password for user hacluster. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully.4、配置backup-mfsmaster
[root@server4 ~]# ls moosefs-master-3.0.103-1.rhsystemd.x86_64.rpm [root@server4 ~]# rpm -ivh moosefs-master-3.0.103-1.rhsystemd.x86_64.rpm warning: moosefs-master-3.0.103-1.rhsystemd.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID cf82adba: NOKEY Preparing... ################################# [100%] Updating / installing... 1:moosefs-master-3.0.103-1.rhsystem################################# [100%] [root@server4 ~]# vim /usr/lib/systemd/system/moosefs-master.service [Unit] Description=MooseFS Master server Wants=network-online.target After=network.target network-online.target [Service] Type=forking ExecStart=/usr/sbin/mfsmaster -a ExecStop=/usr/sbin/mfsmaster stop ExecReload=/usr/sbin/mfsmaster reload PIDFile=/var/lib/mfs/.mfsmaster.lock TimeoutStopSec=1800 TimeoutStartSec=1800 Restart=no [Install] WantedBy=multi-user.target [root@server4 ~]# systemctl daemon-reload配置yum源
[root@server4 ~]# vim /etc/yum.repos.d/rhel7.repo [rhel7.3] name=rhel7.3 baseurl=http://172.25.12.250/rhel7.3 gpgcheck=0 [HighAvailability] name=HighAvailability baseurl=http://172.25.12.250/rhel7.3/addons/HighAvailability gpgcheck=0 [ResilientStorage] name=ResilientStorage baseurl=http://172.25.12.250/rhel7.3/addons/ResilientStorage gpgcheck=0测试yum源，及下载相关软件
[root@server4 ~]# yum repolist [root@server4 ~]# yum install -y pacemaker corosync pcs启动服务，配置hacluster用户密码（主备的密码需要相同）
[root@server4 ~]# systemctl start pcsd [root@server4 ~]# systemctl enable pcsd Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service. [root@server4 ~]# passwd hacluster Changing password for user hacluster. New password: BAD PASSWORD: The password is shorter than 8 characters Retype new password: passwd: all authentication tokens updated successfully.5、开始创建集群
[root@server1 ~]# pcs cluster auth server1 server4 [root@server1 ~]# pcs cluster setup --name mycluster server1 server4 [root@server1 ~]# pcs status nodes Error: error running crm_mon, is pacemaker running? [root@server1 ~]# pcs cluster start --all [root@server1 ~]# pcs status nodes Pacemaker Nodes: Online: server1 server4 Standby: Maintenance: Offline: Pacemaker Remote Nodes: Online: Standby: Maintenance: Offline:验证corosync是否正常：
[root@server1 ~]# corosync-cfgtool -s查看corosync的状态：
[root@server1 ~]# pcs status corosync6、创建主备集群
[root@server1 ~]# pcs property set stonith-enabled=false [root@server1 ~]# crm_verify -L -V [root@server1 ~]# pcs status创建vip：
[root@server1 ~]# pcs resource create vip ocf:heartbeat:IPaddr2 ip=172.25.12.100 cidr_netmask=32 op monitor interval=30s [root@server1 ~]# ip a查看监控：
crm_mon7、执行故障转移
[root@server1 ~]# pcs cluster stop server1 server1: Stopping Cluster (pacemaker)... server1: Stopping Cluster (corosync)...在server4查看监控：
crm_mon当server1的master重新开启时，服务还是在server4上不会抢占资源：
[root@server1 ~]# pcs cluster start server1 server1: Starting Cluster...关闭server4，此时vip移到server1：
[root@server4 ~]# pcs cluster stop server4 server4: Stopping Cluster (pacemaker)... server4: Stopping Cluster (corosync)...实验实现了mfs的高可用，当master出现故障时，资源会自动转移到master-backup，当master恢复后，资源不会回切，保障了客户端服务的稳定性。
获取可用资源标准列表：
[root@server1 ~]# pcs resource standards ocf lsb service systemd查看资源提供者的列表：
[root@server1 ~]# pcs resource providers heartbeat openstack pacemaker查看特定的可用资源代理：
[root@server1 ~]# pcs resource agents ocf:heartbeat1、关闭服务，卸载目录。
客户端：
[root@foundation1 ~]# umount /mnt/mfs root@foundation1 ~]# umount /mnt/mfsmeta/ [root@foundation1 ~]# vim /etc/hosts 172.25.12.100 mfsmasterserver1：
[root@server1 ~]# systemctl stop moosefs-master [root@server1 ~]# vim /etc/hosts 172.25.12.100 mfsmasterserver2：
[root@server2 ~]# systemctl stop moosefs-chunkserver [root@server2 ~]# vim /etc/hosts 172.25.12.100 mfsmasterserver3：
[root@server3 ~]# systemctl stop moosefs-chunkserver [root@server3 ~]# vim /etc/hosts 172.25.12.100 mfsmasterserver4：
[root@server4 ~]# systemctl stop moosefs-master [root@server4 ~]# vim /etc/hosts 172.25.12.100 mfsmaster2、使用server2为iscsi的服务端，添加一块磁盘
检测硬盘是否添加成功：
[root@server2 ~]# fdisk -l3、server2下载iscsi软件，及共享磁盘的配置
[root@server2 ~]# yum install -y targetcli #安装远程快存储设备 [root@server2 ~]# systemctl start target #开启服务 [root@server2 ~]# targetcli Warning: Could not load preferences file /root/.targetcli/prefs.bin. targetcli shell version 2.1.fb41 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'. /> cd backstores/block /backstores/block> create my_disk1 /dev/sdb Created block storage object my_disk1 using /dev/sdb. /backstores/block> cd .. /backstores> cd .. /> cd iscsi /iscsi> create iqn.2019-05.com.example:server2 Created target iqn.2019-05.com.example:server2. Created TPG 1. Global pref auto_add_default_portal=true Created default portal listening on all IPs (0.0.0.0), port 3260. /iscsi> cd iqn.2019-05.com.example:server2 /iscsi/iqn.20...ample:server2> cd tpg1/luns /iscsi/iqn.20...er2/tpg1/luns> create /backstores/block/my_disk1 Created LUN 0. /iscsi/iqn.20...er2/tpg1/luns> cd .. /iscsi/iqn.20...:server2/tpg1> cd acls /iscsi/iqn.20...er2/tpg1/acls> create iqn.2019-05.com.example:client Created Node ACL for iqn.2019-05.com.example:client Created mapped LUN 0. /iscsi/iqn.20...er2/tpg1/acls> cd .. /iscsi/iqn.20...:server2/tpg1> cd .. /iscsi/iqn.20...ample:server2> cd .. /iscsi> cd .. /> exit Global pref auto_save_on_exit=true Last 10 configs saved in /etc/target/backup. Configuration saved to /etc/target/saveconfig.json4、在iscsi的客户端：server1上安装iscsi客户端软件，对共享磁盘进行分区使用
[root@server1 ~]# yum install -y iscsi-* [root@server1 ~]# vim /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2019-05.com.example:client [root@server1 ~]# iscsiadm -m discovery -t st -p 172.25.12.2 172.25.12.2:3260,1 iqn.2019-05.com.example:server2 [root@server1 ~]# iscsiadm -m node -l Logging in to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.12.2,3260] (multiple) Login to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.12.2,3260] successful.[root@server1 ~]# fdisk -l ##查看远程共享出来的磁盘
[root@server1 ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x1c160cb8. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (65528-16777215, default 65528): Using default value 65528 Last sector, +sectors or +size{K,M,G} (65528-16777215, default 16777215): Using default value 16777215 Partition 1 of type Linux and of size 8 GiB is set Command (m for help): p Disk /dev/sdb: 8589 MB, 8589934592 bytes, 16777216 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 33550336 bytes Disk label type: dos Disk identifier: 0x1c160cb8 Device Boot Start End Blocks Id System /dev/sdb1 65528 16777215 8355844 83 Linux Command (m for help): wq The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.5、格式化分区及使用
[root@server1 ~]# mkfs.xfs /dev/sdb1 #格式化分区 [root@server1 ~]# dd if=/dev/zero of=/dev/sdb bs=512 count=1 #破坏分区 [root@server1 ~]# fdisk -l /dev/sdb #查看不到分区因为截取速度太快，导致分区被自动删除，因为把/dev/sda下的所有空间都分给了/dev/sda1，只需要重新建立分区即可。
[root@server1 ~]# mount /dev/sdb1 /mnt # 查看到分区不存在 mount: special device /dev/sdb1 does not exist [root@server1 ~]# fdisk /dev/sdb Command (m for help): n Select (default p): p Partition number (1-4, default 1): First sector (65528-16777215, default 65528): Using default value 65528 Last sector, +sectors or +size{K,M,G} (65528-16777215, default 16777215): Device Boot Start End Blocks Id System /dev/sdb1 65528 16777215 8355844 83 Linux Command (m for help): wq [root@server1 ~]# mount /dev/sdb1 /mnt [root@server1 ~]# df [root@server1 ~]# cd /var/lib/mfs/ # 这是mfs的数据目录 [root@server1 mfs]# ls changelog.1.mfs changelog.5.mfs metadata.mfs metadata.mfs.empty changelog.4.mfs metadata.crc metadata.mfs.back.1 stats.mfs [root@server1 mfs]# cp -p * /mnt # 带权限拷贝/var/lib/mfs的所有数据文件到/dev/sdb1上 [root@server1 mfs]# cd /mnt [root@server1 mnt]# ll [root@server1 mnt]# chown mfs.mfs /mnt # 当目录属于mfs用户和组时，才能正常使用 [root@server1 mnt]# ll -d /mnt drwxr-xr-x 2 mfs mfs 185 May 18 20:55 /mnt [root@server1 mnt]# cd [root@server1 ~]# umount /mnt [root@server1 ~]# mount /dev/sdb1 /var/lib/mfs/ # 使用分区，测试是否可以使用共享磁盘 [root@server1 ~]# df [root@server1 ~]# systemctl start moosefs-master # 服务开启成功，就说明数据文件拷贝成功，共享磁盘可以正常使用 [root@server1 ~]# ps ax测试成功后关闭服务：
[root@server1 ~]# systemctl stop moosefs-master6、配置backup-master，是之也可以使用共享磁盘
[root@server4 ~]# vim /etc/hosts 172.25.12.1 server1 mfsmaster [root@server4 ~]# yum install -y iscsi-* [root@server4 ~]# vim /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.2019-05.com.example:client [root@server4 ~]# iscsiadm -m discovery -t st -p 172.25.12.2 172.25.12.2:3260,1 iqn.2019-05.com.example:server2 [root@server4 ~]# iscsiadm -m node -l Logging in to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.12.2,3260] (multiple) Login to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.12.2,3260] successful. [root@server4 ~]# fdisk -l [root@server4 ~]# mount /dev/sdb1 /var/lib/mfs/ [root@server4 ~]# systemctl start moosefs-master [root@server4 ~]# systemctl stop moosefs-master [root@server4 ~]# pcs cluster start server4 server4: Starting Cluster...7、在master上创建mfs文件系统
[root@server1 ~]# pcs resource create mfsdata ocf:heartbeat:Filesystem device=/dev/sdb1 directory=/var/lib/mfs fstype=xfs op monitor interval=30s [root@server1 ~]# pcs resource show vip (ocf::heartbeat:IPaddr2): Started server4 mfsdata (ocf::heartbeat:Filesystem): Started server1 crm_monvip和mfsdata在不同主机。
[root@server1 ~]# pcs resource create mfsd systemd:moosefs-master op monitor interval=1min 创建mfsd系统 [root@server1 ~]# pcs resource group add mfsgroup vip mfsdata mfsd # 把vip，mfsdata，mfsd 集中在一个组中 [root@server1 ~]# pcs cluster stop server1 # 当关闭master之后，master上的服务就会迁移到backup-master上 server1: Stopping Cluster (pacemaker)... server1: Stopping Cluster (corosync)...在server4上查：
crm_mon三、fence解决脑裂问题 （1）先在客户端测试高可用 先在客户端测试高可用
[root@server2 ~]# systemctl start moosefs-chunkserver [root@server2 ~]# ping mfsmaster # 保证解析可以通信[root@server3 ~]# systemctl start moosefs-chunkserver [root@server3 ~]# ping mfsmaster # 保证解析可以通信 查看vip的位置
[root@server4 ~]# ip a开启master
[root@server1 ~]# pcs cluster start server1 server1: Starting Cluster...在客户端进行分布式存储测试
[root@foundation1 ~]# cd /mnt/mfs/ [root@foundation1 mfs]# ls [root@foundation1 mfs]# mfsmount #有可能挂载失败，是因为这个目录不为空 mfsmaster accepted connection with parameters: read-write,restricted_ip,admin ; root mapped to root:root [root@foundation1 mfs]# mkdir dir{1..2} [root@foundation1 mfs]# ls dir1 dir2 [root@foundation1 mfs]# cd /mnt/mfs/dir1/ [root@foundation1 dir1]# dd if=/dev/zero of=file2 bs=1M count=2000 # 我们上传一份大文件 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 10.9616 s, 191 MB/s [root@server4 ~]# pcs cluster stop server4 # 在客户端上传大文件的同时，关闭正在提供服务的服务端 server4: Stopping Cluster (pacemaker)... server4: Stopping Cluster (corosync)... [root@foundation1 dir1]# mfsfileinfo file2 # 我们查看到文件上传成功，并没有受到影响当master挂掉之后，backup-master会立刻接替master的工作，保证客户端可以进行正常访问，但是，当master重新运行时，master可能会抢回自己的工作，从而导致master和backup-master同时修改同一份数据文件从而发生脑裂，此时使用fence。
（2）安装fence服务
[root@server1 ~]# yum install fence-virt -y [root@server1 ~]# mkdir /etc/cluster [root@server4 ~]# yum install fence-virt -y [root@server4 ~]# mkdir /etc/cluster（3）生成一份fence密钥文件，传给服务端
[root@foundation1 ~]# yum install -y fence-virtd [root@foundation1 ~]# yum install fence-virtd-libvirt -y [root@foundation1 ~]# yum install fence-virtd-multicast -y [root@foundation1 ~]# fence_virtd -c Listener module [multicast]: Multicast IP Address [225.0.0.12]: Multicast IP Port [1229]: Interface [br0]: br0 Key File [/etc/cluster/fence_xvm.key]: Backend module [libvirt]:[root@foundation1 ~]# mkdir /etc/cluster # 这是存放密钥的文件，需要自己手动建立 [root@foundation1 ~]# dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=128 count=1 [root@foundation1 ~]# systemctl start fence_virtd [root@foundation1 ~]# cd /etc/cluster/ [root@foundation1 cluster]# ls fence_xvm.key [root@foundation1 cluster]# scp fence_xvm.key root@172.25.12.1:/etc/cluster/ [root@foundation1 cluster]# scp fence_xvm.key root@172.25.12.4:/etc/cluster/
查看域名 [root@foundation1 cluster]# virsh list 查看域名
查看是否匹配域名。
在master查看监控crm_mon这是主机名。与主机所对应的域名一致。
[root@server1 ~]# cd /etc/cluster/ [root@server1 cluster]# pcs stonith create vmfence fence_xvm pcmk_host_map=":server1;server4:server4" op monitor interval=1min [root@server1 cluster]# pcs property set stonith-enabled=true [root@server1 cluster]# crm_verify -L -V [root@server1 cluster]# fence_xvm -H server4 # 使server4断电重启 [root@server1 cluster]# crm_mon # 查看监控，server4上的服务迁移到master的server1上 [root@server1 cluster]# echo c > /proc/sysrq-trigger # 模拟master端内核崩溃![在这里插入图片描述]查看监控，server4会立刻接管master的所有服务 server1重启成功后，不会抢占资源，不会出现脑裂的状况。 查看监控发现，master重启成功之后，并不会抢占资源，服务依旧在backup-master端正常运行，说明fence生效。
