部署kubernates集群环境
部署环境准备集群类型采用多对多高可用集群部署,共7台主机,3台master,3台slaver,1台client。
主机名
OS版本
ip
主机配置
备注
region-master-1
7.6.1160
192.168.199.130
2颗CPU4G内存
region-master-2
7.6.1160
192.168.199.131
2颗CPU4G内存
region-master-3
7.6.1160
192.168.199.132
2颗CPU4G内存
region-slaver-1
7.6.1160
192.168.199.180
2颗CPU4G内存
region-slaver-2
7.6.1160
192.168.199.181
2颗CPU4G内存
region-slaver-3
7.6.1160
192.168.199.182
2颗CPU4G内存
region-vip
7.6.1160
192.168.199.188
2颗CPU4G内存
region-client
7.6.1160
192.168.199.160
2颗CPU4G内存
系统环境准备分别在master和slaver节点都执行下面操作。
配置操作系统
禁用了防火墙和selinux并设置了阿里源。
$systemctlstopfirewalldsystemctldisablefirewalld
$setenforce0
$vim/etc/selinux/config
SELINUX=disabled
配置主机名修改主机名[root@localhost~]more/etc/hostname
退出重新登陆即可显示新设置的主机名region-master-1
修改hosts文件[root@region-master-1~]swapoff-a
永久禁用禁用swap后还需修改配置文件/etc/fstab,注释swap
[root@region-master-1~]/'/etc/fstab


本文的k8s网络使用flannel,该网络需要设置内核参数bridge-nf-call-iptables=1,修改这个参数需要系统有br_netfilter模块。
br_netfilter模块加载查看br_netfilter模块:
[root@region-master-1~]modprobebr_netfilter
该方式重启后会失效
永久新增br_netfilter模块:
[root@region-master-1~]!/bin/bash
forfilein/etc/sysconfig/modules/*.modules;do
[-x$file]$file
done
EOF
[root@region-master-1~]chmod755/etc/sysconfig/modules/br_
内核参数临时修改[root@region-master-1~]=1
=1
内核参数永久修改[root@region-master-1~]sysctl-p/etc//
=1
=1
新增kubernetes源[root@region-master-1~]yumcleanall
[root@region-master-1~]ssh-keygen-trsa
将秘钥同步至region-master-2/region-master-3[root@region-master-1~]ssh-copy-id-i/root/.ssh/id_@172.27.34.5
免密登陆测试[root@region-master-1~]sshregion-master-3
region-master-1可以直接登录region-master-2和region-master-3,不需要输入密码。
Docker安装controlplane和work节点都执行本部分操作。
安装依赖包[root@region-master-1~]yum-config-manager--add-repo
安装DockerCEdocker安装版本查看[root@region-master-1~]
启动Docker[root@region-master-1~]systemctlenabledocker
命令补全安装bash-completion[root@region-master-1~]source/etc//bash_
镜像加速由于DockerHub的服务器在国外,下载镜像会比较慢,可以配置镜像加速器。主要的加速器有:Docker官方提供的中国registrymirror、阿里云加速器、DaoCloud加速器,本文以阿里加速器配置为例。
登陆阿里云容器模块登陆地址为:
配置镜像加速器配置文件
[root@region-master-1~]tee/etc/docker/'EOF'
{
"registry-mirrors":[""]
}
EOF
重启服务
[root@region-master-1~]systemctlrestartdocker
加速器配置完成
验证[root@region-master-1~]dockerrunhello-world
通过查询docker版本和运行容器hello-world来验证docker是否安装成功。
修改CgroupDriver修改修改,新增‘”exec-opts”:[“=systemd”’
[root@region-master-1~]systemctldaemon-reload
[root@region-master-1~]yum-yinstallkeepalived
keepalived配置region-master-1上keepalived配置:
[root@region-master-1~]more/etc/keepalived/
!ConfigurationFileforkeepalived
global_defs{
router_idregion-master-2
}
vrrp_instanceVI_1{
stateBACKUP
interfaceens160
virtual_router_id50
priority90
advert_int1
authentication{
auth_typePASS
auth_pass1111
}
virtual_ipaddress{
192.168.199.188
}
}
region-master-3上keepalived配置:
[root@region-master-3~]servicekeepalivedstart
[root@region-master-1~]ipa
vip在region-master-1上
k8s安装controlplane和work节点都执行本部分操作。
版本查看[root@region-master-1~]
调整Kubernetes仓库
vi/etc//
[kubernetes]
name=Kubernetes
baseurl=
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=
刷新仓库
yumcleanall
yummakecache
启动kubelet启动kubelet并设置开机启动
[root@region-master-1~]echo"source(kubectlcompletionbash)"~/.bash_profile
[root@region-master-1~]
./
[root@region-master-1~]
apiVersion:/v1beta2
kind:ClusterConfiguration
kubernetesVersion:
apiServer:
certSANs:kubeadminit--config=
记录kubeadmjoin的输出,后面需要这个命令将work节点和其他master节点加入集群中。
Youcannowjoinanynumberofcontrol-planenodesbycopyingcertificateauthorities
andserviceaccountkeysoneachnodeandthenrunningthefollowingasroot:
:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966\
--control-plane
Thenyoucanjoinanynumberofworkernodesbyrunningthefollowingoneachasroot:
:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966
初始化失败:
如果初始化失败,可执行kubeadmreset后重新初始化
[root@region-master-1~]rm-rf$HOME/.kube/config
加载环境变量[root@region-master-1~]_profile
本文所有操作都在root用户下执行,若为非root用户,则执行如下操作:
mkdir-p$HOME/.kube
cp-i/etc/kubernetes/$HOME/.kube/config
chown$(id-u):$(id-g)$HOME/.kube/config
安装flannel网络在region-master-1上新建flannel网络
[root@region-master-1~]ll|
-rwxr--r--1rootroot6381月215:23
[root@region-master-1~]customizable
CONTROL_PLANE_IPS="172.27.34.4172.27.34.5"
forhostin${CONTROL_PLANE_IPS};do
scp/etc/kubernetes/pki/"${USER}"@$host:
scp/etc/kubernetes/pki/"${USER}"@$host:
scp/etc/kubernetes/pki/"${USER}"@$host:
scp/etc/kubernetes/pki/"${USER}"@$host:
scp/etc/kubernetes/pki/"${USER}"@$host:
scp/etc/kubernetes/pki/"${USER}"@$host:
scp/etc/kubernetes/pki/etcd/"${USER}"@$host:
pwd
/root
[root@region-master-2~]
USER=rootQuotethislineifyouareusingexternaletcd
mv/${USER}//etc/kubernetes/pki/etcd/
[root@region-master-2~]pwd
/root
[root@region-master-3~]./
region-master-2加入集群:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966\
--control-plane
region-master-3加入集群:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966\
--control-plane
加载环境变量region-master-2和region-master-3加载环境变量
[root@region-master-2~]echo"exportKUBECONFIG=/etc/kubernetes/"~/.bash_profile
[root@region-master-2~]scpregion-master-1:/etc/kubernetes//etc/kubernetes/
[root@region-master-3~]_profile
该步操作是为了在region-master-2和region-master-3上也能执行kubectl命令。
集群节点查看[root@region-master-1~]kubectlgetpo-owide-nkube-system
所有master节点处于ready状态,所有的系统组件也正常。
:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966
运行初始化master生成的work节点加入集群的命令
region-slaver-2加入集群:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966
region-slaver-3加入集群:6443--\
--discovery-token-ca-cert-hashsha256:e306ffc7a126eb1f2c0cab297bbbed04f5bb464a04c05f1b0171192acbbae966
集群节点查看[root@region-master-1~]catEOF/etc//
[kubernetes]
name=Kubernetes
baseurl=
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=
EOF
更新缓存[root@client~]yum-ymakecache
安装kubectl[root@client~]yum-yinstallbash-completion
加载bash-completion[root@client~]mkdir-p/etc/kubernetes
[root@client~]echo"exportKUBECONFIG=/etc/kubernetes/"~/.bash_profile
[root@client~]echo"source(kubectlcompletionbash)"~/.bash_profile
[root@region-master-1~]kubectlgetnodes
[root@client~]kubectlgetpo-owide-nkube-system
Dashboard搭建本节内容都在client端完成
下载yaml[root@client~]sed-i's/kubernetesui/\/loong576/g'
由于默认的镜像仓库网络访问不通,故改成阿里镜像
2.2外网访问
[root@client~]
---
apiVersion:v1
kind:ServiceAccount
metadata:
name:dashboard-admin
namespace:kubernetes-dashboard
---
apiVersion:/v1beta1
kind:ClusterRoleBinding
metadata:
name:dashboard-admin
subjects:
-kind:ServiceAccount
name:dashboard-admin
namespace:kubernetes-dashboard
roleRef:
apiGroup:
kind:ClusterRole
name:cluster-admin
部署访问部署Dashboard[root@client~]kubectlgetall-nkubernetes-dashboard
令牌查看[root@client~]ipa|grep130
/32scopeglobalens160
[root@client~]kubectlgetpointskube-scheduler-nkube-system-oyaml|grepholderIdentity
/leader:'{"holderIdentity":"region-master-1_720d65f9-e425-4058-95d7-e5478ac951f7","leaseDurationSeconds":15,"acquireTime":"2020-01-02T09:36:20Z","renewTime":"2020-01-03T07:58:03Z","leaderTransitions":2}'

[root@region-master-1~]ipa|grep130
/32scopeglobalens160
controller-manager和scheduler也发生了迁移
[root@client~]kubectlgetpointskube-scheduler-nkube-system-oyaml|grepholderIdentity
/leader:'{"holderIdentity":"region-master-3_e0a2ec66-c415-44ae-871c-18c73258dc8f","leaseDurationSeconds":15,"acquireTime":"2020-01-03T08:04:56Z","renewTime":"2020-01-03T08:06:45Z","leaderTransitions":3}'
集群功能性测试查询:
[root@client~]
apiVersion:apps/v1创建资源类型为Deployment
metadata:Deployment名称
spec:指定副本数为3
template:定义Pod的元数据
labels:label的key和value分别为app和nginx
spec:容器的名称
image:nginx:
/nginx-mastercreated
[root@client~]init0
查看VIP[root@region-master-3~]kubectlgetnodes
Errorfromserver:etcdserver:requesttimedout
[root@client~]#kubectlgetnodes
:6443wasrefused-didyouspecifytherighthostorport?
etcd集群崩溃,整个k8s集群也不能正常对外服务。