部署kubernates集群环境

部署环境准备

集群类型采用多对多高可用集群部署,共7台主机,3台master,3台slaver,1台client。

主机名

OS版本

ip

主机配置

备注

region-master-1

7.6.1160

192.168.199.130

2颗CPU4G内存

region-master-2

7.6.1160

192.168.199.131

2颗CPU4G内存

region-master-3

7.6.1160

192.168.199.132

2颗CPU4G内存

region-slaver-1

7.6.1160

192.168.199.180

2颗CPU4G内存

region-slaver-2

7.6.1160

192.168.199.181

2颗CPU4G内存

region-slaver-3

7.6.1160

192.168.199.182

2颗CPU4G内存

region-vip

7.6.1160

192.168.199.188

2颗CPU4G内存

region-client

7.6.1160

192.168.199.160

2颗CPU4G内存

系统环境准备

分别在master和slaver节点都执行下面操作。

配置操作系统

禁用了防火墙和selinux并设置了阿里源。

$systemctlstopfirewalldsystemctldisablefirewalld

$setenforce0

$vim/etc/selinux/config

SELINUX=disabled

配置主机名修改主机名

[root@localhost~]more/etc/hostname

退出重新登陆即可显示新设置的主机名region-master-1

修改hosts文件

[root@region-master-1~]swapoff-a

永久禁用

禁用swap后还需修改配置文件/etc/fstab,注释swap

[root@region-master-1~]/'/etc/fstab



内核参数修改

本文的k8s网络使用flannel,该网络需要设置内核参数bridge-nf-call-iptables=1,修改这个参数需要系统有br_netfilter模块。

br_netfilter模块加载

查看br_netfilter模块:

[root@region-master-1~]modprobebr_netfilter

该方式重启后会失效

永久新增br_netfilter模块:

[root@region-master-1~]!/bin/bash

forfilein/etc/sysconfig/modules/*.modules;do

[-x$file]$file

done

EOF

[root@region-master-1~]chmod755/etc/sysconfig/modules/br_

内核参数临时修改

[root@region-master-1~]=1

=1

内核参数永久修改

[root@region-master-1~]sysctl-p/etc//

=1

=1

新增kubernetes源

[root@region-master-1~]yumcleanall

[root@region-master-1~]ssh-keygen-trsa

将秘钥同步至region-master-2/region-master-3

[root@region-master-1~]ssh-copy-id-i/root/.ssh/id_@172.27.34.5

免密登陆测试

[root@region-master-1~]sshregion-master-3

region-master-1可以直接登录region-master-2和region-master-3,不需要输入密码。

Docker安装

controlplane和work节点都执行本部分操作。

安装依赖包

[root@region-master-1~]yum-config-manager--add-repo

安装DockerCEdocker安装版本查看

[root@region-master-1~]

启动Docker

[root@region-master-1~]systemctlenabledocker

命令补全安装bash-completion

[root@region-master-1~]source/etc//bash_

镜像加速

由于DockerHub的服务器在国外,下载镜像会比较慢,可以配置镜像加速器。主要的加速器有:Docker官方提供的中国registrymirror、阿里云加速器、DaoCloud加速器,本文以阿里加速器配置为例。

登陆阿里云容器模块

登陆地址为:

配置镜像加速器

配置文件

[root@region-master-1~]tee/etc/docker/'EOF'

{

"registry-mirrors":[""]

}

EOF

重启服务

[root@region-master-1~]systemctlrestartdocker

加速器配置完成

验证

[root@region-master-1~]dockerrunhello-world

通过查询docker版本和运行容器hello-world来验证docker是否安装成功。

修改CgroupDriver修改

修改,新增‘”exec-opts”:[“=systemd”’

[root@region-master-1~]systemctldaemon-reload

[root@region-master-1~]yum-yinstallkeepalived

keepalived配置

region-master-1上keepalived配置:

[root@region-master-1~]more/etc/keepalived/

!ConfigurationFileforkeepalived

global_defs{

router_idregion-master-2

}

vrrp_instanceVI_1{

stateBACKUP

interfaceens160

virtual_router_id50

priority90

advert_int1

authentication{

auth_typePASS

auth_pass1111

}

virtual_ipaddress{

192.168.199.188

}

}

region-master-3上keepalived配置:

[root@region-master-3~]servicekeepalivedstart

[root@region-master-1~]ipa

vip在region-master-1上

k8s安装

controlplane和work节点都执行本部分操作。

版本查看

[root@region-master-1~]

调整Kubernetes仓库

vi/etc//

[kubernetes]

name=Kubernetes

baseurl=

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=

刷新仓库

yumcleanall

yummakecache

启动kubelet

启动kubelet并设置开机启动

[root@region-master-1~]echo"source(kubectlcompletionbash)"~/.bash_profile

[root@region-master-1~]

./

[root@region-master-1~]

apiVersion:/v1beta2

kind:ClusterConfiguration

kubernetesVersion:

apiServer:

certSANs:kubeadminit--config=

记录kubeadmjoin的输出,后面需要这个命令将work节点和其他master节点加入集群中。

Youcannowjoinanynumberofcontrol-planenodesbycopyingcertificateauthorities

andserviceaccountkeysoneachnodeandthenrunningthefollowingasroot:

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966\

--control-plane

Thenyoucanjoinanynumberofworkernodesbyrunningthefollowingoneachasroot:

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966

初始化失败:

如果初始化失败,可执行kubeadmreset后重新初始化

[root@region-master-1~]rm-rf$HOME/.kube/config

加载环境变量

[root@region-master-1~]_profile

本文所有操作都在root用户下执行,若为非root用户,则执行如下操作:

mkdir-p$HOME/.kube

cp-i/etc/kubernetes/$HOME/.kube/config

chown$(id-u):$(id-g)$HOME/.kube/config

安装flannel网络

在region-master-1上新建flannel网络

[root@region-master-1~]ll|

-rwxr--r--1rootroot6381月215:23

[root@region-master-1~]customizable

CONTROL_PLANE_IPS="172.27.34.4172.27.34.5"

forhostin${CONTROL_PLANE_IPS};do

scp/etc/kubernetes/pki/"${USER}"@$host:

scp/etc/kubernetes/pki/"${USER}"@$host:

scp/etc/kubernetes/pki/"${USER}"@$host:

scp/etc/kubernetes/pki/"${USER}"@$host:

scp/etc/kubernetes/pki/"${USER}"@$host:

scp/etc/kubernetes/pki/"${USER}"@$host:

scp/etc/kubernetes/pki/etcd/"${USER}"@$host:

pwd

/root

[root@region-master-2~]

USER=rootQuotethislineifyouareusingexternaletcd

mv/${USER}//etc/kubernetes/pki/etcd/

[root@region-master-2~]pwd

/root

[root@region-master-3~]./

region-master-2加入集群

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966\

--control-plane

region-master-3加入集群

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966\

--control-plane

加载环境变量

region-master-2和region-master-3加载环境变量

[root@region-master-2~]echo"exportKUBECONFIG=/etc/kubernetes/"~/.bash_profile

[root@region-master-2~]scpregion-master-1:/etc/kubernetes//etc/kubernetes/

[root@region-master-3~]_profile

该步操作是为了在region-master-2和region-master-3上也能执行kubectl命令。

集群节点查看

[root@region-master-1~]kubectlgetpo-owide-nkube-system


所有master节点处于ready状态,所有的系统组件也正常。

Slave节点加入集群region-slaver-1加入集群

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966

运行初始化master生成的work节点加入集群的命令

region-slaver-2加入集群

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966

region-slaver-3加入集群

:6443--\

--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966

集群节点查看

[root@region-master-1~]catEOF/etc//

[kubernetes]

name=Kubernetes

baseurl=

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=

EOF

更新缓存

[root@client~]yum-ymakecache

安装kubectl

[root@client~]yum-yinstallbash-completion

加载bash-completion

[root@client~]mkdir-p/etc/kubernetes

[root@client~]echo"exportKUBECONFIG=/etc/kubernetes/"~/.bash_profile

[root@client~]echo"source(kubectlcompletionbash)"~/.bash_profile

[root@region-master-1~]kubectlgetnodes

[root@client~]kubectlgetpo-owide-nkube-system

Dashboard搭建

本节内容都在client端完成

下载yaml

[root@client~]sed-i's/kubernetesui/\/loong576/g'

由于默认的镜像仓库网络访问不通,故改成阿里镜像

2.2外网访问

[root@client~]

---

apiVersion:v1

kind:ServiceAccount

metadata:

name:dashboard-admin

namespace:kubernetes-dashboard

---

apiVersion:/v1beta1

kind:ClusterRoleBinding

metadata:

name:dashboard-admin

subjects:

-kind:ServiceAccount

name:dashboard-admin

namespace:kubernetes-dashboard

roleRef:

apiGroup:

kind:ClusterRole

name:cluster-admin

部署访问部署Dashboard

[root@client~]kubectlgetall-nkubernetes-dashboard

令牌查看

[root@client~]ipa|grep130

/32scopeglobalens160

[root@client~]kubectlgetpointskube-scheduler-nkube-system-oyaml|grepholderIdentity

/leader:'{"holderIdentity":"region-master-1_720d65f9-e425-4058-95d7-e5478ac951f7","leaseDurationSeconds":15,"acquireTime":"2020-01-02T09:36:20Z","renewTime":"2020-01-03T07:58:03Z","leaderTransitions":2}'


region-master-1关机关闭region-master-1

[root@region-master-1~]ipa|grep130

/32scopeglobalens160

controller-manager和scheduler也发生了迁移

[root@client~]kubectlgetpointskube-scheduler-nkube-system-oyaml|grepholderIdentity

/leader:'{"holderIdentity":"region-master-3_e0a2ec66-c415-44ae-871c-18c73258dc8f","leaseDurationSeconds":15,"acquireTime":"2020-01-03T08:04:56Z","renewTime":"2020-01-03T08:06:45Z","leaderTransitions":3}'

集群功能性测试

查询:

[root@client~]

apiVersion:apps/v1创建资源类型为Deployment

metadata:Deployment名称

spec:指定副本数为3

template:定义Pod的元数据

labels:label的key和value分别为app和nginx

spec:容器的名称

image:nginx:

/nginx-mastercreated

[root@client~]init0

查看VIP

[root@region-master-3~]kubectlgetnodes

Errorfromserver:etcdserver:requesttimedout

[root@client~]#kubectlgetnodes

:6443wasrefused-didyouspecifytherighthostorport?

etcd集群崩溃,整个k8s集群也不能正常对外服务。