docker集群-kubernetes集群化配置

发表于 | 分类于 | 评论数： | 阅读次数：
本文字数： 29k | 阅读时长 ≈ 26 分钟

摘要

本文部分内容来源于网络，个人收集整理，请勿传播

时间跨越有点久了，之前的实验和文章都没有写完，最近重新开始趟这个坑。

docker安装

请移步 docker安装学习记录

kubernetes

本节讲以两种方式安装kubernets（一下简称k8s）集群；

  • 使用kubeadm创建集群
  • 使用二进制包创建集群

由于k8s集群搭建过程中需要去google的仓库拉取一些镜像，而这个仓库在国内是访问不到的，所以可以使用以下两种解决办法

第一种方式搭建集群比较简单，主要服务都是以容器的方式运行的;

第二种方式搭建起来麻烦一点，优点是可以了解整个集群的组件架构；

下面的内容是两种搭建方式公用的部分。

使用repo安装 kubectl、kubelet and kubeadm

centos7

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 阿里源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF

# 官方源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

setenforce 0
yum install -y kubectl kubelet kubeadm

ubuntu

1
2
3
4
5
6
7
apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

修改内核参数

1
2
3
4
5
6
vim /etc/sysctl.conf

net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

环境配置

1
2
3
4
rm -r -f /etc/kubernetes /var/lib/kubelet /var/lib/etcd;
echo "10.0.0.176.master" > /etc/hostname 
echo "127.0.0.1   10.0.0.176.master" >> /etc/hosts
sysctl kernel.hostname=10.0.0.176.master

使用kubeadm构建k8s集群

修改配置

1
2
3
4
5
6
7
8
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# 把 systemd 改成 cgroupfs

KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs
# 注意这里一定要先启动一次，
systemctl enable kubelet && systemctl start kubelet
# 查看启动状态，这里就算启动失败也是无所谓的
systemctl status kubelet

初始化集群

1
2
3
4
5
kubeadm init \
	--apiserver-advertise-address=0.0.0.0 \
	--pod-network-cidr 10.244.0.0/16

## --pod-network-cidr参数，如果要使用flannel等管理网络一定要加这个

集群初始化会去拉取镜像，如果已经提前处理好，正常执行几分钟就应该初始化完成。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.7.1
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12
[preflight] WARNING: docker service is not enabled, please run 'systemctl enable docker.service'
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [bjo-ep-dep-039.dev.fwmrm.net kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.17.139]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 31.001311 seconds
[token] Using token: 472def.6bbb304791b76492
[apiconfig] Created RBAC rules
[addons] Applied essential addon: kube-proxy
[addons] Applied essential addon: kube-dns

Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run (as a regular user):
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node as root:
kubeadm join --token e5698b.f0ad65ab9db2a951 10.0.0.176:6443

配置config

根据提示执行以下命令

1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Master Isolation

由于安全原因，默认情况下pod不会被schedule到master节点上，可以通过下面命令解除这种限制

1
kubectl taint nodes --all node-role.kubernetes.io/master-

添加node节点

1
2
3
4
5
6
# 可以查看集群token
kubeadm token list
TOKEN                     TTL         EXPIRES   USAGES                   DESCRIPTION
e7d41c.e3faa005ab37c4cf   <forever>   <never>   authentication,signing   The default bootstrap token generated by 'kubeadm init'.

kubeadm join --token e7d41c.e3faa005ab37c4cf 10.0.0.176:6443 --skip-preflight-checks

检查状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kubectl get node
NAME                STATUS    AGE       VERSION
10.0.0.176.master   Ready     9h        v1.7.2
10.0.0.177.node     Ready     9h        v1.7.2


kubectl get po -o wide -n kube-system
NAME                                        READY     STATUS    RESTARTS   AGE       IP            NODE
etcd-10.0.0.176.master                      1/1       Running   2          9h        10.0.0.176    10.0.0.176.master
kube-apiserver-10.0.0.176.master            1/1       Running   2          9h        10.0.0.176    10.0.0.176.master
kube-controller-manager-10.0.0.176.master   1/1       Running   3          9h        10.0.0.176    10.0.0.176.master
kube-dns-2425271678-bz6n6                   0/3       Pending   6          9h        10.244.0.93   10.0.0.176.master
kube-proxy-bx6qb                            1/1       Running   1          9h        10.0.0.177    a10.0.0.177.node
kube-proxy-pt73w                            1/1       Running   2          9h        10.0.0.176    10.0.0.176.master
kube-scheduler-10.0.0.176.master            1/1       Running   3          9h        10.0.0.176    10.0.0.176.master

此时dns应该处于Pending状态，是因为没有安装管理pod的网络模块，如果没有可控制的node节点也会处于此状态。

安装flannel管理pod网络

1
2
3
4
5
6
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# weave
wget https://git.io/weave-kube -O weave-kube.yaml

查看状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
kubectl get po -o wide -n kube-system
NAME                                        READY     STATUS    RESTARTS   AGE       IP            NODE
etcd-10.0.0.176.master                      1/1       Running   2          9h        10.0.0.176    10.0.0.176.master
kube-apiserver-10.0.0.176.master            1/1       Running   2          9h        10.0.0.176    10.0.0.176.master
kube-controller-manager-10.0.0.176.master   1/1       Running   3          9h        10.0.0.176    10.0.0.176.master
kube-dns-2425271678-bz6n6                   3/3       Running   6          9h        10.244.0.93   10.0.0.176.master
kube-flannel-ds-08mfq                       2/2       Running   8          9h        10.0.0.176    10.0.0.176.master
kube-flannel-ds-jn41h                       2/2       Running   3          9h        10.0.0.177    a10.0.0.177.node
kube-proxy-bx6qb                            1/1       Running   1          9h        10.0.0.177    a10.0.0.177.node
kube-proxy-pt73w                            1/1       Running   2          9h        10.0.0.176    10.0.0.176.master
kube-scheduler-10.0.0.176.master            1/1       Running   3          9h        10.0.0.176    10.0.0.176.master

kubectl describe svc kube-dns -n kube-system
Name:			kube-dns
Namespace:		kube-system
Labels:			k8s-app=kube-dns
			kubernetes.io/cluster-service=true
			kubernetes.io/name=KubeDNS
Annotations:		<none>
Selector:		k8s-app=kube-dns
Type:			ClusterIP
IP:			10.96.0.10
Port:			dns	53/UDP
Endpoints:		10.244.0.93:53
Port:			dns-tcp	53/TCP
Endpoints:		10.244.0.93:53
Session Affinity:	None
Events:			<none>

修改配置

1
2
3
4
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

# --cluster-dns=10.244.0.79 修改成 --cluster-dns=10.244.0.93
# 重启kubelet

kube-dashboard

获取yml文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
curl https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml > kubernetes-dashboard.yaml

# 修改一些配置，配置nodeport

# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.6 (RBAC enabled).
#
# Example usage: kubectl create -f <this_file>

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
      - name: kubernetes-dashboard
        image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          # - --apiserver-host=http://my-address:port
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 9090
    nodePort: 31001
  selector:
    k8s-app: kubernetes-dashboard

启动

1
2
3
4
5
kubectl create -f kubernetes-dashboard.yaml

https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml
https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb
kubectl create -f deploy/kube-config/influxdb

Kubernetes 1.6开始API Server启用了RBAC授权，kubernetes-dashboard.yaml中并未定义需要授权的ServiceAccount，如果访问Dashborad：http://ClusterIP:NodePort会被拒绝。

ServiceAccount的配置以后再说，目前还是有一些问题没有解决

使用二进制文件部署k8s

etcd集群安装

本文使用静态配置的方式搭建，详细操作以及介绍请移步 etcd-cluster 部署

安装etcd

1
2
3
4
5
6
7
8
9
ETCD_VER="v3.2.4"
DOWNLOAD_URL="https://github.com/coreos/etcd/releases/download"
LOCAL_DIR="/data/soft"
mkdir ${LOCAL_DIR}
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o ${LOCAL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz
mkdir -p ${LOCAL_DIR}/etcd && tar xzvf ${LOCAL_DIR}/etcd-${ETCD_VER}-linux-amd64.tar.gz -C ${LOCAL_DIR}/etcd --strip-components=1

cd ${LOCAL_DIR}/etcd/ && cp etcd etcdctl /usr/local/bin
etcd -version

静态配置

静态配置主要预先将集群的配置信息分配好，然后将集群分布启动，集群将根据配置信息组成集群。这里按如下的配置信息分别启动三个etcd。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
mkdir /data/bin -p && cd /data/bin && touch etcd.sh
# 基本配置 在每个节点都要添加
NAME_1=infra1
NAME_2=infra2
NAME_3=infra3
HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13

ETCD_DATA_DIR="/data/infra.etcd/"
# 这里需要根据不同节点修改
# ETCD_NAME="${NAME_1}"
# ETCD_IP="${HOST_1}"

ETCD_LISTEN_PEER_URLS="http://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="http://${ETCD_IP}:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://${ETCD_IP}":2380
ETCD_INITIAL_CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-test"
ETCD_ADVERTISE_CLIENT_URLS="http://${ETCD_IP}:2379"

pkill etcd
etcd --name ${ETCD_NAME} --data-dir ${ETCD_DATA_DIR} \
    --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
    --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS} \
    --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
    --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
    --initial-cluster ${ETCD_INITIAL_CLUSTER} \
    --initial-cluster-state ${ETCD_INITIAL_CLUSTER_STATE} \
    --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN}

配置supervisor

详情请移步 etcd-cluster 部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
mkdir -p /etc/supervisor/conf.d/
mkdir -p /data/logs/supervisor/

vim /etc/supervisor/supervisord.conf

[unix_http_server]
file=/var/run/supervisor.sock   ; (the path to the socket file)
chmod=0700                       ; sockef file mode (default 0700)

[supervisord]
logfile=/data/logs/supervisor/supervisord.log ; (main log file;default $CWD/supervisord.log)
pidfile=/var/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
childlogdir=/data/logs/supervisor            ; ('AUTO' child log dir, default $TEMP)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket

[include]
files = /etc/supervisor/conf.d/*.conf

#########################################

vim /etc/supervisor/conf.d/etcd.conf

[program:etcd]
directory = /data/bin/
command = /bin/sh etcd.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/etcd.log
stderr_logfile = /data/logs/supervisor/etcd_err.log

supervisord -c /etc/supervisor/supervisord.conf

环境变量配置

本文最开始使用的是etcd2接口，目前etcd3接口变化比较大，所以现在直接重新按照etcd3的接口进行配置

1
2
3
4
5
6
7
8
9
10
11
cd /data/bin && vim etcd.env

#export ETCDCTL_API=3
export ETCDCTL_API=2
export HOST_1=172.16.1.16
export HOST_2=172.16.1.17
export HOST_3=172.16.1.13
export ENDPOINTS=http://$HOST_1:2379,http://$HOST_2:2379,http://$HOST_3:2379
alias etcdctl='etcdctl --endpoints=$ENDPOINTS'

source /data/bin/etcd.env

集群验证

按如上配置分别启动集群，启动集群后，将会进入集群选举状态，若出现大量超时，则需要检查主机的防火墙是否关闭，或主机之间是否能通过2380端口通信，集群建立后通过以下命令检查集群状态。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
etcdctl member list -w table
+------------------+---------+--------+-------------------------+-------------------------+
|        ID        | STATUS  |  NAME  |       PEER ADDRS        |      CLIENT ADDRS       |
+------------------+---------+--------+-------------------------+-------------------------+
| cd1adc80e78ec1cf | started | infra3 | http://172.16.1.13:2380 | http://172.16.1.13:2379 |
| e74b186f7d620f76 | started | infra2 | http://172.16.1.17:2380 | http://172.16.1.17:2379 |
| ea91822d14c58cfb | started | infra1 | http://172.16.1.16:2380 | http://172.16.1.16:2379 |
+------------------+---------+--------+-------------------------+-------------------------+

etcdctl cluster-health

etcdctl endpoint health
172.16.1.13:2379 is healthy: successfully committed proposal: took = 1.683776ms
172.16.1.17:2379 is healthy: successfully committed proposal: took = 2.115015ms
172.16.1.16:2379 is healthy: successfully committed proposal: took = 2.494229ms

在其中一个节点执行

1
2
3
4
5
6
7
8
# api2
etcdctl mkdir /k8s/network
etcdctl set /k8s/network/config '{"network":"172.100.0.0/16"}'
{"network":"172.100.0.0/16"}

# api3
etcdctl put /k8s/network/config '{"network":"172.100.0.0/16"}'
OK

其他节点执行有正常返回结果说明集群搭建成功

1
2
3
etcdctl get /k8s/network/config
/k8s/network/config
{"network":"172.100.0.0/16"}

该命令含义是，期望docker运行的container实例的地址，都在 172.100.0.0/16网段中

flanneld会读取/k8s/network目录中config值，然后接管docker的地址分配，并把docker和宿主机器之间的网络桥接起来。

flannel

安装flannel

1
2
3
4
5
6
7
FLANNEL_VERSION="v0.8.0"
LOCAL_DIR="/data/soft"
FLANNEL_URL="https://github.com/coreos/flannel/releases/download"
mkdir ${LOCAL_DIR}
cd ${LOCAL_DIR} && wget ${FLANNEL_URL}/${FLANNEL_VERSION}/flannel-${FLANNEL_VERSION}-linux-amd64.tar.gz
tar xvf flannel-${FLANNEL_VERSION}-linux-amd64.tar.gz
cp flanneld /usr/local/bin && flanneld -version

启动flannel

1
2
3
4
5
6
7
8
9
10
11
cd /data/bin && touch flanneld.sh

HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13
ENDPOINTS=http://$HOST_1:2379,http://$HOST_2:2379,http://$HOST_3:2379

pkill flanneld
flanneld \
	-etcd-endpoints=${ENDPOINTS} \
	-etcd-prefix=/k8s/network

supervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
vim /etc/supervisor/conf.d/flanneld.conf
[program:flanneld]
directory = /data/bin/
command = /bin/sh flanneld.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/flanneld.log
stderr_logfile = /data/logs/supervisor/flanneld_err.log

supervisorctl update
supervisorctl status
etcd                             RUNNING   pid 30867, uptime 0:34:13
flanneld                         RUNNING   pid 41079, uptime 0:00:25

docker配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
source /run/flannel/subnet.env

cat <<E0F> /etc/docker/daemon.json
{
    "max-concurrent-downloads": 3,
    "max-concurrent-uploads": 3,
    "disable-legacy-registry": true,
    "insecure-registries": [
        "http://r.isme.pub"
    ],
    "registry-mirrors": [
        "https://uss7pbj4.mirror.aliyuncs.com"
    ],
    "log-driver": "json-file",
    "log-opts": {
        "labels": "io.cass.isme.pub,log.ignore",
        "max-size": "1g",
        "max-file": "10"
    },
    "graph": "/data/docker",
    "storage-driver": "overlay",
    "bip": "${FLANNEL_SUBNET}",
    "mtu": ${FLANNEL_MTU},
    "hosts": [
        "tcp://127.0.0.1:4243",
        "unix:///var/run/docker.sock"
    ]
}
E0F

rc=0
ip link show docker0 >/dev/null 2>&1 || rc="$?"
if [[ "$rc" -eq "0" ]]; then
ip link set dev docker0 down
ip link delete docker0
fi


systemctl daemon-reload
systemctl enable docker
systemctl restart docker

安装k8s

1
2
3
4
5
6
7
8
9
10
KUB_VERSION="v1.7.2"
LOCAL_DIR="/data/soft"
K8S_URL="https://github.com/kubernetes/kubernetes/releases/download"
mkdir ${LOCAL_DIR}
cd ${LOCAL_DIR} && wget ${K8S_URL}/${KUB_VERSION}/kubernetes.tar.gz
tar xvf kubernetes.tar.gz && cd kubernetes/cluster
./get-kube-binaries.sh
cd ../server/ && tar xvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin
cp kube-apiserver kube-controller-manager kube-proxy kube-scheduler /usr/local/bin/

启动k8s集群

Master

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
mkdir /data/logs/k8s/{apiserver,controller-manager,scheduler} -p
cd /data/bin && touch kube.env kube-api.sh kube-con.sh kube-sche.sh

HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13
ENDPOINTS=http://$HOST_1:2379,http://$HOST_2:2379,http://$HOST_3:2379
LOG_DIR="/data/logs/k8s"

source /data/bin/kube.env
THIS_IP=${HOST_1}
pkill kube-apiserver
kube-apiserver \
	--insecure-bind-address=0.0.0.0 \
	--insecure-port=8080 \
	--etcd_servers=${ENDPOINTS} \
	--logtostderr=false \
	--allow-privileged=false \
	--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,SecurityContextDeny,ResourceQuota \
	--service-node-port-range=30000-32767 \
	--advertise-address=${THIS_IP} \
	--log_dir=${LOG_DIR}/apiserver \
	--service-cluster-ip-range=172.100.0.0/16

source /data/bin/kube.env
THIS_IP=${HOST_1}
pkill kube-controller-manager
kube-controller-manager \
	--master=${THIS_IP}:8080 \
	--enable-hostpath-provisioner=false \
	--logtostderr=false \
	--log_dir=${LOG_DIR}/controller-manager

source /data/bin/kube.env
THIS_IP=${HOST_1}
pkill kube-scheduler
kube-scheduler --master=${THIS_IP}:8080 \
	--logtostderr=false \
	--log_dir=${LOG_DIR}/scheduler

master-supervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
cd /etc/supervisor/conf.d/ && touch kube-api.conf kube-con.conf kube-sche.conf

vim kube-api.conf
[program:kube-api]
directory = /data/bin/
command = /bin/sh kube-api.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-api.log
stderr_logfile = /data/logs/supervisor/kube-api_err.log

vim kube-con.conf
[program:kube-con]
directory = /data/bin/
command = /bin/sh kube-con.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-con.log
stderr_logfile = /data/logs/supervisor/kube-con_err.log

vim kube-sche.conf
[program:kube-sche]
directory = /data/bin/
command = /bin/sh kube-sche.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-sche.log
stderr_logfile = /data/logs/supervisor/kube-sche_err.log

supervisorctl update
kube-api: added process group
kube-con: added process group
kube-sche: added process group

supervisorctl status
etcd                             RUNNING   pid 30867, uptime 2:11:59
flanneld                         RUNNING   pid 41079, uptime 1:38:11
kube-api                         RUNNING   pid 41543, uptime 0:00:04
kube-con                         RUNNING   pid 41544, uptime 0:00:04
kube-sche                        RUNNING   pid 41547, uptime 0:00:04

node

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
mkdir /data/logs/k8s/{kubelet,proxy} -p
cd /data/bin && touch kube.env kubelet.sh kube-proxy.sh

HOST_1=172.16.1.16
HOST_2=172.16.1.17
HOST_3=172.16.1.13

MASTER=${HOST_1}
THIS_IP=${HOST_2}
LOG_DIR="/data/logs/k8s"

source /data/bin/kube.env
pkill kubelet
kubelet \
	--address=0.0.0.0 \
	--port=10250 \
	--log_dir=${LOG_DIR}/kubelet \
	--hostname_override=${THIS_IP} \
	--api_servers=http://${MASTER}:8080 \
	--logtostderr=false

source /data/bin/kube.env
pkill kube-proxy
kube-proxy \
	--master=${MASTER}:8080 \
	--logtostderr=false

node-supervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
cd /etc/supervisor/conf.d/ && touch kubelet.conf kube-proxy.conf

vim kubelet.conf
[program:kubelet]
directory = /data/bin/
command = /bin/sh kubelet.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kubelet.log
stderr_logfile = /data/logs/supervisor/kubelet_err.log

vim kube-proxy.conf
[program:kube-proxy]
directory = /data/bin/
command = /bin/sh kube-proxy.sh
user = root
autostart = true
autorestart = true
stdout_logfile = /data/logs/supervisor/kube-proxy.log
stderr_logfile = /data/logs/supervisor/kube-proxy_err.log


supervisorctl update
kube-proxy: added process group
kubelet: added process group

supervisorctl status
etcd                             RUNNING   pid 30890, uptime 2:25:32
flanneld                         RUNNING   pid 41025, uptime 1:56:50
kube-proxy                       RUNNING   pid 41805, uptime 0:01:34
kubelet                          RUNNING   pid 41742, uptime 0:01:38

配置skydns

1
etcdctl mk /skydns/config '{"dns-addr":"172.100.81.2:53","ttl":3600,"domain":"sky."}'

附录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
docker容器所依赖的六种namespace级别的资源隔离技术：

1.UTS(UNIX Time-sharing system)
  允许创建独立的hostname/domian_name namespace，作为独立的节点
2.IPC(Inter Process Communication)
  创建隔离的进程间通信方式，实现包括信号量、消息队列、管道、共享内存等资源的namespace隔离
3.PID(Process ID)
  允许树的子叶节点namespace创建自己专属的pid区间，pid对同级节点无效，仅父级namespace节点可见。(在子节点的pid，会在父节点映射为另一个pid标志，
  因此在父节点的角度，每个子节点的进程拥有两个pid号)
4.mount
  挂载点namespace，各个namespace默认的mount挂载只在本空间内生效，不影响其他namespace，也可使用share共享挂载的方式互相影响， 或者slave从属挂
  载的方式，实现主影响从。
5.network
  每个net ns拥有独立的网络资源(包括网络设备,IPv4/IPv6协议栈，防火墙，/proc/net、/sys/class/net目录等)
6.user
  每个user ns拥有独立的id、属性、key、根目录，组织结构等
  

docker:
  - docker ps -a
  - docker build -t xxx .  #根据Dockerfile生成image
  - docker images 
  - docker exec -it {ID\NAME} /bin/bash | /bin/sh
  - docker run -it {ID\NAME} -p -v 
  - docker start {ID\NAME}
  - docker stop {ID\NAME}
  - docker rm {ID\NAME}
  - docker rmi {ID\NAME}
  - docker save coredns/coredns:1.0.0 | gzip > coredns.tar.gz  #将已有的img打包
  - docker load -i IMAGE #docker载入本地image打包文件
  - docker stats {ID\NAME}  #查看容器资源占用情况，不填查看全部
  - docker cp
  - docker commit -p -m 'xxx' 1a889d5bbf99 xxx  #将容器打标签提交为img，-p选项意思是打包时暂停容器，不会退出
  - docker push xxx  #push到容器仓库
  

k8s:
  - kubectl get pods -o wide 
  - kubectl get pod xxx -o yaml  #获取yaml配置文件
  - kubectl get nodes -o wide 
  - kubectl set image deployment xxx xxx=image_url  #更改部署镜像
  - kubectl describe pod mysql-deploy-766bd7dbcb-7nxvw  #查看pod的创建运行记录
  - kubectl scale deploy/kube-dns --replicas=3    #修改deploy的副本数
  - kubectl create -f xxx.yaml  #创建资源
  - kubectl delete deploy mysql-deploy   #删除资源
  - kubectl get svc -o wide
  - kubectl get ep SVC_NAME   #查看svc对应绑定的pod
  - kubectl get rs
  - kubectl get deploy/DEPLOY-NAME
  - kubectl get all  #获取所有类型资源
  - kubectl get componentstatuses   #获取k8s各组件健康状态，简写为kubectl get cs
  - kubectl describe deploy/DEPLOY-NAME
  - kubectl status rollout deploy/DEPLOY-NAME
  - kubectl label nodes 171 disktype=ssd            #添加标签
  - kubectl label nodes 171 disktype-               #删除标签
  - kubectl label nodes 171 disktype=hdd --overwrite #修改标签
  - kubectl logs POD_NAME  #查看pod的日志，排错用
  - kubectl get nodes -o json | jq '.items[] | .spec'  #查看每个node的CIDR分配
  - kubectl delete pod NAME --grace-period=0 --force   #强制删除资源，在1.3版本去掉--force选项
  - kubectl replace -f xxx.yaml   #更改定义资源的yaml配置文件
  - kubectl get secret -n kube-system | grep dashboard  #查找secret
  - kubectl describe secret -n kube-system kubernetes-dashboard-token-ld92d  #查看该secret的令牌
  - kubectl scale --replicas=3 deployment/xxxx   #横向扩展deploy的rs数量
  - kubectl cordon NODENAME   #将node设置为检修状态，不再向此node调度新的pod
  - kubectl drain NODENAME    #将node设置为（排水）不可用状态，并且驱逐其上的pod转移至其他正常node上。这一步会进行两个步骤：1.将node设为cordon状态2.驱逐node上的pod
  - kubectl drain node2 --delete-local-data --force --ignore-daemonsets  #忽略ds，否则存在ds的话无法正常执行
  - kubectl uncordon NODENAME  #取消检修状态、排水状态
  - kubectl proxy --address=0.0.0.0 --port=8001 --accept-hosts=^.* &  #kubectl监听端口代理apiserver的服务，允许调用客户端免去认证步骤
  - kubectl patch deployment my-nginx --patch '{"spec": {"template": {"metadata": {"annotations": {"update-time": "2018-04-11 12:15" }}}}}'   #更新configmap之后挂载configmap 的pod中的环境变量不会自动更新，可以通过更新一下deployment.spec.template的注解内容来触发pod的滚动更新。


**k8s restful api**
#### GET方法
##  labelSelector
# 获取标签{app: pimsystemdev}的pod
http://192.168.9.27:8001/api/v1/namespaces/default/pods?labelSelector=app%3Dpimsystemdev

# 动态获取标签{app: pimsystemdev}的pod
http://192.168.9.27:8001/api/v1/namespaces/default/pods?watch&labelSelector=app%3Dpimsystemdev

## fieldSelector
# 根据level 1字段获取信息
http://192.168.9.27:8001/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dykstaskdev-657c7f56fc-7vnd4