docker-kubernetes 集群化配置

摘要

时间跨越有点久了,之前的实验和文章都没有写完,最近重新开始趟这个坑。

docker安装

请移步 docker安装学习记录

kubernetes

本节讲以两种方式安装kubernets(一下简称k8s)集群;

  • 使用kubeadm创建集群
  • 使用二进制包创建集群

由于k8s集群搭建过程中需要去google的仓库拉取一些镜像,而这个仓库在国内是访问不到的,所以可以使用以下两种解决办法

第一种方式搭建集群比较简单,主要服务都是以容器的方式运行的;

第二种方式搭建起来麻烦一点,优点是可以了解整个集群的组件架构;

下面的内容是两种搭建方式公用的部分。

使用repo安装 kubectl、kubelet and kubeadm

centos7

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 阿里源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
# 官方源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubectl kubelet kubeadm

ubuntu

1
2
3
4
5
6
7
apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

修改内核参数

1
2
3
4
5
6
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

环境配置

1
2
3
4
rm -r -f /etc/kubernetes /var/lib/kubelet /var/lib/etcd;
echo "10.0.0.176.master" > /etc/hostname
echo "127.0.0.1 10.0.0.176.master" >> /etc/hosts
sysctl kernel.hostname=10.0.0.176.master

使用kubeadm构建k8s集群

修改配置

1
2
3
4
5
6
7
8
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# 把 systemd 改成 cgroupfs
KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs
# 注意这里一定要先启动一次,
systemctl enable kubelet && systemctl start kubelet
# 查看启动状态,这里就算启动失败也是无所谓的
systemctl status kubelet

初始化集群

1
2
3
4
5
kubeadm init \
--apiserver-advertise-address=0.0.0.0 \
--pod-network-cidr 10.244.0.0/16
## --pod-network-cidr参数,如果要使用flannel等管理网络一定要加这个

集群初始化会去拉取镜像,如果已经提前处理好,正常执行几分钟就应该初始化完成。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.1
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12
[preflight] WARNING: docker service is not enabled, please run 'systemctl enable docker.service'
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [bjo-ep-dep-039.dev.fwmrm.net kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.17.139]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 31.001311 seconds
[token] Using token: 472def.6bbb304791b76492
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/
You can now join any number of machines by running the following on each node as root:
kubeadm join --token e5698b.f0ad65ab9db2a951 10.0.0.176:6443

配置config

根据提示执行以下命令

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Master Isolation

由于安全原因,默认情况下pod不会被schedule到master节点上,可以通过下面命令解除这种限制

1
kubectl taint nodes --all node-role.kubernetes.io/master-

添加node节点

1
2
3
4
5
6
# 可以查看集群token
kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION
e7d41c.e3faa005ab37c4cf <forever> <never> authentication,signing The default bootstrap token generated by 'kubeadm init'.
kubeadm join --token e7d41c.e3faa005ab37c4cf 10.0.0.176:6443 --skip-preflight-checks

检查状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kubectl get node
NAME STATUS AGE VERSION
10.0.0.176.master Ready 9h v1.7.2
10.0.0.177.node Ready 9h v1.7.2
kubectl get po -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE
etcd-10.0.0.176.master 1/1 Running 2 9h 10.0.0.176 10.0.0.176.master
kube-apiserver-10.0.0.176.master 1/1 Running 2 9h 10.0.0.176 10.0.0.176.master
kube-controller-manager-10.0.0.176.master 1/1 Running 3 9h 10.0.0.176 10.0.0.176.master
kube-dns-2425271678-bz6n6 0/3 Pending 6 9h 10.244.0.93 10.0.0.176.master
kube-proxy-bx6qb 1/1 Running 1 9h 10.0.0.177 a10.0.0.177.node
kube-proxy-pt73w 1/1 Running 2 9h 10.0.0.176 10.0.0.176.master
kube-scheduler-10.0.0.176.master 1/1 Running 3 9h 10.0.0.176 10.0.0.176.master

此时dns应该处于Pending状态,是因为没有安装管理pod的网络模块,如果没有可控制的node节点也会处于此状态。

安装flannel管理pod网络

1
2
3
4
5
6
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
# weave
wget https://git.io/weave-kube -O weave-kube.yaml

查看状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
kubectl get po -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE
etcd-10.0.0.176.master 1/1 Running 2 9h 10.0.0.176 10.0.0.176.master
kube-apiserver-10.0.0.176.master 1/1 Running 2 9h 10.0.0.176 10.0.0.176.master
kube-controller-manager-10.0.0.176.master 1/1 Running 3 9h 10.0.0.176 10.0.0.176.master
kube-dns-2425271678-bz6n6 3/3 Running 6 9h 10.244.0.93 10.0.0.176.master
kube-flannel-ds-08mfq 2/2 Running 8 9h 10.0.0.176 10.0.0.176.master
kube-flannel-ds-jn41h 2/2 Running 3 9h 10.0.0.177 a10.0.0.177.node
kube-proxy-bx6qb 1/1 Running 1 9h 10.0.0.177 a10.0.0.177.node
kube-proxy-pt73w 1/1 Running 2 9h 10.0.0.176 10.0.0.176.master
kube-scheduler-10.0.0.176.master 1/1 Running 3 9h 10.0.0.176 10.0.0.176.master
kubectl describe svc kube-dns -n kube-system
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations: <none>
Selector: k8s-app=kube-dns
Type: ClusterIP
IP: 10.96.0.10
Port: dns 53/UDP
Endpoints: 10.244.0.93:53
Port: dns-tcp 53/TCP
Endpoints: 10.244.0.93:53
Session Affinity: None
Events: <none>

修改配置

1
2
3
4
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# --cluster-dns=10.244.0.79 修改成 --cluster-dns=10.244.0.93
# 重启kubelet

kube-dashboard

获取yml文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
curl https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml > kubernetes-dashboard.yaml
# 修改一些配置,配置nodeport
# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.6 (RBAC enabled).
#
# Example usage: kubectl create -f <this_file>
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
containers:
- name: kubernetes-dashboard
image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3
ports:
- containerPort: 9090
protocol: TCP
args:
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
livenessProbe:
httpGet:
path: /
port: 9090
initialDelaySeconds: 30
timeoutSeconds: 30
serviceAccountName: kubernetes-dashboard
# Comment the following tolerations if Dashboard must not be deployed on master
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 80
targetPort: 9090
nodePort: 31001
selector:
k8s-app: kubernetes-dashboard

启动

1
2
3
4
5
kubectl create -f kubernetes-dashboard.yaml
https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml
https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb
kubectl create -f deploy/kube-config/influxdb

Kubernetes 1.6开始API Server启用了RBAC授权,kubernetes-dashboard.yaml中并未定义需要授权的ServiceAccount,如果访问Dashborad:http://ClusterIP:NodePort会被拒绝。

ServiceAccount的配置以后再说,目前还是有一些问题没有解决

使用二进制文件部署k8s

etcd集群安装

本文使用静态配置的方式搭建,详细操作以及介绍请移步 etcd-cluster 部署

安装etcd

1
2
3
4
5
6
7
8
9
ETCD_VER="v3.2.4"
DOWNLOAD_URL="https://github.com/coreos/etcd/releases/download"
LOCAL_DIR="/data/soft"
mkdir ${LOCAL_DIR}
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o ${LOCAL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz
mkdir -p ${LOCAL_DIR}/etcd && tar xzvf ${LOCAL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz -C ${LOCAL_DIR}/etcd --strip-components=1
cd ${LOCAL_DIR}/etcd/ && cp etcd etcdctl /usr/local/bin
etcd -version

静态配置

静态配置主要预先将集群的配置信息分配好,然后将集群分布启动,集群将根据配置信息组成集群。这里按如下的配置信息分别启动三个etcd。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
mkdir /data/bin -p && cd /data/bin && touch etcd.sh
# 基本配置 在每个节点都要添加
NAME_1=infra1
NAME_2=infra2
NAME_3=infra3
HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13
ETCD_DATA_DIR="/data/infra.etcd/"
# 这里需要根据不同节点修改
# ETCD_NAME="${NAME_1}"
# ETCD_IP="${HOST_1}"
ETCD_LISTEN_PEER_URLS="http://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="http://${ETCD_IP}:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://${ETCD_IP}":2380
ETCD_INITIAL_CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-test"
ETCD_ADVERTISE_CLIENT_URLS="http://${ETCD_IP}:2379"
pkill etcd
etcd --name ${ETCD_NAME} --data-dir ${ETCD_DATA_DIR} \
--listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls ${ETCD_LISTEN_CLIENT_URLS} \
--initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-cluster ${ETCD_INITIAL_CLUSTER} \
--initial-cluster-state ${ETCD_INITIAL_CLUSTER_STATE} \
--initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN}

配置supervisor

详情请移步 etcd-cluster 部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
mkdir -p /etc/supervisor/conf.d/
mkdir -p /data/logs/supervisor/
vim /etc/supervisor/supervisord.conf
[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
chmod=0700 ; sockef file mode (default 0700)
[supervisord]
logfile=/data/logs/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/data/logs/supervisor ; ('AUTO' child log dir, default $TEMP)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
[include]
files = /etc/supervisor/conf.d/*.conf
#########################################
vim /etc/supervisor/conf.d/etcd.conf
[program:etcd]
directory = /data/bin/
command = /bin/sh etcd.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/etcd.log
stderr_logfile = /data/logs/supervisor/etcd_err.log
supervisord -c /etc/supervisor/supervisord.conf

环境变量配置

本文最开始使用的是etcd2接口,目前etcd3接口变化比较大,所以现在直接重新按照etcd3的接口进行配置

1
2
3
4
5
6
7
8
9
10
11
cd /data/bin && vim etcd.env
#export ETCDCTL_API=3
export ETCDCTL_API=2
export HOST_1=172.16.1.16
export HOST_2=172.16.1.17
export HOST_3=172.16.1.13
export ENDPOINTS=http://$HOST_1:2379,http://$HOST_2:2379,http://$HOST_3:2379
alias etcdctl='etcdctl --endpoints=$ENDPOINTS'
source /data/bin/etcd.env

集群验证

按如上配置分别启动集群,启动集群后,将会进入集群选举状态,若出现大量超时,则需要检查主机的防火墙是否关闭,或主机之间是否能通过2380端口通信,集群建立后通过以下命令检查集群状态。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
etcdctl member list -w table
+------------------+---------+--------+-------------------------+-------------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+--------+-------------------------+-------------------------+
| cd1adc80e78ec1cf | started | infra3 | http://172.16.1.13:2380 | http://172.16.1.13:2379 |
| e74b186f7d620f76 | started | infra2 | http://172.16.1.17:2380 | http://172.16.1.17:2379 |
| ea91822d14c58cfb | started | infra1 | http://172.16.1.16:2380 | http://172.16.1.16:2379 |
+------------------+---------+--------+-------------------------+-------------------------+
etcdctl cluster-health
etcdctl endpoint health
172.16.1.13:2379 is healthy: successfully committed proposal: took = 1.683776ms
172.16.1.17:2379 is healthy: successfully committed proposal: took = 2.115015ms
172.16.1.16:2379 is healthy: successfully committed proposal: took = 2.494229ms

在其中一个节点执行

1
2
3
4
5
6
7
8
# api2
etcdctl mkdir /k8s/network
etcdctl set /k8s/network/config '{"network":"172.100.0.0/16"}'
{"network":"172.100.0.0/16"}
# api3
etcdctl put /k8s/network/config '{"network":"172.100.0.0/16"}'
OK

其他节点执行有正常返回结果说明集群搭建成功

1
2
3
etcdctl get /k8s/network/config
/k8s/network/config
{"network":"172.100.0.0/16"}

该命令含义是,期望docker运行的container实例的地址,都在 172.100.0.0/16网段中

flanneld会读取/k8s/network目录中config值,然后接管docker的地址分配,并把docker和宿主机器之间的网络桥接起来。

flannel

安装flannel

1
2
3
4
5
6
7
FLANNEL_VERSION="v0.8.0"
LOCAL_DIR="/data/soft"
FLANNEL_URL="https://github.com/coreos/flannel/releases/download"
mkdir ${LOCAL_DIR}
cd ${LOCAL_DIR} && wget ${FLANNEL_URL}/${FLANNEL_VERSION}/flannel-${FLANNEL_VERSION}-linux-amd64.tar.gz
tar xvf flannel-${FLANNEL_VERSION}-linux-amd64.tar.gz
cp flanneld /usr/local/bin && flanneld -version

启动flannel

1
2
3
4
5
6
7
8
9
10
11
cd /data/bin && touch flanneld.sh
HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13
ENDPOINTS=http://$HOST_1:2379,http://$HOST_2:2379,http://$HOST_3:2379
pkill flanneld
flanneld \
-etcd-endpoints=${ENDPOINTS} \
-etcd-prefix=/k8s/network

supervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
vim /etc/supervisor/conf.d/flanneld.conf
[program:flanneld]
directory = /data/bin/
command = /bin/sh flanneld.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/flanneld.log
stderr_logfile = /data/logs/supervisor/flanneld_err.log
supervisorctl update
supervisorctl status
etcd RUNNING pid 30867, uptime 0:34:13
flanneld RUNNING pid 41079, uptime 0:00:25

docker配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
source /run/flannel/subnet.env
cat <<E0F> /etc/docker/daemon.json
{
"max-concurrent-downloads": 3,
"max-concurrent-uploads": 3,
"disable-legacy-registry": true,
"insecure-registries": [
"http://r.isme.pub"
],
"registry-mirrors": [
"https://uss7pbj4.mirror.aliyuncs.com"
],
"log-driver": "json-file",
"log-opts": {
"labels": "io.cass.isme.pub,log.ignore",
"max-size": "1g",
"max-file": "10"
},
"graph": "/data/docker",
"storage-driver": "overlay",
"bip": "${FLANNEL_SUBNET}",
"mtu": ${FLANNEL_MTU},
"hosts": [
"tcp://127.0.0.1:4243",
"unix:///var/run/docker.sock"
]
}
E0F
rc=0
ip link show docker0 >/dev/null 2>&1 || rc="$?"
if [[ "$rc" -eq "0" ]]; then
ip link set dev docker0 down
ip link delete docker0
fi
systemctl daemon-reload
systemctl enable docker
systemctl restart docker

安装k8s

1
2
3
4
5
6
7
8
9
10
KUB_VERSION="v1.7.2"
LOCAL_DIR="/data/soft"
K8S_URL="https://github.com/kubernetes/kubernetes/releases/download"
mkdir ${LOCAL_DIR}
cd ${LOCAL_DIR} && wget ${K8S_URL}/${KUB_VERSION}/kubernetes.tar.gz
tar xvf kubernetes.tar.gz && cd kubernetes/cluster
./get-kube-binaries.sh
cd ../server/ && tar xvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-controller-manager kube-proxy kube-scheduler /usr/local/bin/

启动k8s集群

Master

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
mkdir /data/logs/k8s/{apiserver,controller-manager,scheduler} -p
cd /data/bin && touch kube.env kube-api.sh kube-con.sh kube-sche.sh
HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13
ENDPOINTS=http://$HOST_1:2379,http://$HOST_2:2379,http://$HOST_3:2379
LOG_DIR="/data/logs/k8s"
source /data/bin/kube.env
THIS_IP=${HOST_1}
pkill kube-apiserver
kube-apiserver \
--insecure-bind-address=0.0.0.0 \
--insecure-port=8080 \
--etcd_servers=${ENDPOINTS} \
--logtostderr=false \
--allow-privileged=false \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,SecurityContextDeny,ResourceQuota \
--service-node-port-range=30000-32767 \
--advertise-address=${THIS_IP} \
--log_dir=${LOG_DIR}/apiserver \
--service-cluster-ip-range=172.100.0.0/16
source /data/bin/kube.env
THIS_IP=${HOST_1}
pkill kube-controller-manager
kube-controller-manager \
--master=${THIS_IP}:8080 \
--enable-hostpath-provisioner=false \
--logtostderr=false \
--log_dir=${LOG_DIR}/controller-manager
source /data/bin/kube.env
THIS_IP=${HOST_1}
pkill kube-scheduler
kube-scheduler --master=${THIS_IP}:8080 \
--logtostderr=false \
--log_dir=${LOG_DIR}/scheduler

master-supervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
cd /etc/supervisor/conf.d/ && touch kube-api.conf kube-con.conf kube-sche.conf
vim kube-api.conf
[program:kube-api]
directory = /data/bin/
command = /bin/sh kube-api.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-api.log
stderr_logfile = /data/logs/supervisor/kube-api_err.log
vim kube-con.conf
[program:kube-con]
directory = /data/bin/
command = /bin/sh kube-con.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-con.log
stderr_logfile = /data/logs/supervisor/kube-con_err.log
vim kube-sche.conf
[program:kube-sche]
directory = /data/bin/
command = /bin/sh kube-sche.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-sche.log
stderr_logfile = /data/logs/supervisor/kube-sche_err.log
supervisorctl update
kube-api: added process group
kube-con: added process group
kube-sche: added process group
supervisorctl status
etcd RUNNING pid 30867, uptime 2:11:59
flanneld RUNNING pid 41079, uptime 1:38:11
kube-api RUNNING pid 41543, uptime 0:00:04
kube-con RUNNING pid 41544, uptime 0:00:04
kube-sche RUNNING pid 41547, uptime 0:00:04

node

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
mkdir /data/logs/k8s/{kubelet,proxy} -p
cd /data/bin && touch kube.env kubelet.sh kube-proxy.sh
HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13
MASTER=${HOST_1}
THIS_IP=${HOST_2}
LOG_DIR="/data/logs/k8s"
source /data/bin/kube.env
pkill kubelet
kubelet \
--address=0.0.0.0 \
--port=10250 \
--log_dir=${LOG_DIR}/kubelet \
--hostname_override=${THIS_IP} \
--api_servers=http://${MASTER}:8080 \
--logtostderr=false
source /data/bin/kube.env
pkill kube-proxy
kube-proxy \
--master=${MASTER}:8080 \
--logtostderr=false

node-supervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cd /etc/supervisor/conf.d/ && touch kubelet.conf kube-proxy.conf
vim kubelet.conf
[program:kubelet]
directory = /data/bin/
command = /bin/sh kubelet.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kubelet.log
stderr_logfile = /data/logs/supervisor/kubelet_err.log
vim kube-proxy.conf
[program:kube-proxy]
directory = /data/bin/
command = /bin/sh kube-proxy.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-proxy.log
stderr_logfile = /data/logs/supervisor/kube-proxy_err.log
supervisorctl update
kube-proxy: added process group
kubelet: added process group
supervisorctl status
etcd RUNNING pid 30890, uptime 2:25:32
flanneld RUNNING pid 41025, uptime 1:56:50
kube-proxy RUNNING pid 41805, uptime 0:01:34
kubelet RUNNING pid 41742, uptime 0:01:38

配置skydns

1
etcdctl mk /skydns/config '{"dns-addr":"172.100.81.2:53","ttl":3600,"domain":"sky."}'