前言

尝到k8s甜头以后，我们就想着应用到生产环境里去，以提高业务迭代效率，可是部署在生产环境里有一个要求，就是k8s集群不可以存在单点故障。。。诶唷我的乖乖，这不就要求k8s集群高可用吗，好，下面就是介绍两种目前比较火的k8s集群master高可用方式。

介绍

首先介绍的第一种k8sHA集群我觉得更应该叫做主从结构k8s集群，它由三台master组成，有三个keepalived提供一个vip 来作为apiserver的ip入口，keepalived设置权重，使得vip落在权重大的master节点上，node节点通过访问这个vip从而访问到这一台master，另外两台master则通过etcd集群，来完成数据同步。

缺点：这样的集群是通过keepalived来实现高可用的，也就是说在权重较大的节点没有故障之前，keepalived所指向的流量永远都是经过主master，只有当主master出现故障或者宕机的情况下，才有可能转移到另外两台从master节点上。这样会导致主master节点压力过大，而另外两台从master可能永远不会被调用，导致资源浪费等等情况。

不过，这也是排除单点故障的一种方式。

下面是理想的高可用架构图。

k8s 理想HA高可用

本文中要部署高可用的架构图：

本文高可用架构

上图摘抄至https://www.kubernetes.org.cn/3536.html

好了，到此我们整理一下本文中需要使用的技术栈

keepalived+etcd+k8s master

其中keepalived提供vip供node做apiserver入口，etcd必须是高可用集群，实现数据同步；以及基本的k8s master节点部署。

安装准备

节点部署相关情况

软件版本：

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

以上软件在上一篇初阶k8s集群搭建里已经介绍并附有下载地址。

环境配置

systemctl stop firewalldsystemctl disable firewalld

修改每个节点hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

keepalived安装

libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

wget http://www.keepalived.org/software/keepalived-1.4.3tar.gz

yum install -y libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

yum -y install libnl libnl-devel

tar -xzvf keepalived-1.4.3.tar.gz

cd keepalived-1.4.3

./configure --prefix=/usr/local/keepalived #检查环境配置

出现上图即为正确环境，如果出现错误

checking openssl/ssl.h usability... no

checking openssl/ssl.h presence... no

checking foropenssl/ssl.h... no

configure: error:

!!! OpenSSL is not properly installed on your system. !!!

!!! Can not include OpenSSL headers files. !!!

则：安装openssl和openssl-devel包，然后从新编译配置文件。

yum install openssl openssl-devel

./configure --prefix=/usr/local/keepalived

make && make install

cp keepalived/etc/init.d/keepalived /etc/init.d/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

ps -aux |grep keepalived

chkconfig keepalived on

通过systemctl status keepalived查看keepalived状态

三台master重复以上步骤，直到完成keepalived的安装。

安装完成后编写配置文件：

master1的keepalived.conf

cat >/etc/keepalived/keepalived.conf <<EOF

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state MASTER

interface ens33 #本机物理网卡名字,可通过ip a来查看

virtual_router_id 61

priority 120 # 主节点权重最高依次减少

advert_int 1

mcast_src_ip 192.168.100.1 #修改为本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#注释掉本地IP

#192.168.100.1

192.168.100.2

192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

}

EOF

master2的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP

interface ens33 #本机物理网卡名字,可通过ip a来查看

virtual_router_id 61

priority 110 # 主节点权重最高依次减少

advert_int 1

mcast_src_ip 192.168.100.2 #修改为本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#注释掉本地IP

192.168.100.1

#192.168.100.2

192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

}

EOF

master3的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP

interface ens33 #本机物理网卡名字,可通过ip a来查看

virtual_router_id 61

priority 100 # 主节点权重最高依次减少

advert_int 1

mcast_src_ip 192.168.100.3 #修改为本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#注释掉本地IP

192.168.100.1

192.168.100.2

#192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

}

EOF

启动keepalived

systemctl restart keepalived

通过ip a可以查看

除了本机ip还多了一个虚拟ip

也可以通过ping ip去验证vip是否生效。

安装ETCD

1：设置cfssl环境

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

chmod +x cfssljson_linux-amd64

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

chmod +x cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfoexport PATH=/usr/local/bin:$PATH

2：创建 CA 配置文件（下面配置的IP为etc节点的IP

mkdir /root/ssl

cd /root/ssl

cat > ca-config.json <<EOF

{"signing": {"default": { "expiry": "8760h"},"profiles": { "kubernetes-Soulmate": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" }}}}EOF

cat > ca-csr.json <<EOF

{"CN": "kubernetes-Soulmate","key": {"algo": "rsa","size": 2048},"names": [{ "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System"}]}EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF

{ "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.100.1", "192.168.100.2", "192.168.100.3" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System" } ]}EOF

cfssl gencert -ca=ca.pem

-ca-key=ca-key.pem

-config=ca-config.json

-profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

3:master1分发etcd证书到master2、master3上面

mkdir -p /etc/etcd/ssl

cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/

ssh -n master2 "mkdir -p /etc/etcd/ssl && exit"

ssh -n master3 "mkdir -p /etc/etcd/ssl && exit"

scp -r /etc/etcd/ssl#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

然后执行在master上留下的kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315即可。

作者：我的橙子很甜
链接：https://www.jianshu.com/p/3caccaf8aed1

ITPub博客

高阶k8s HA 集群搭建（一）

前言

介绍

安装准备

keepalived安装

master1的keepalived.conf

master2的keepalived.conf

master3的keepalived.conf

安装ETCD

最新文章