K8s 部署笔记

  1. 环境部署
    1. 配置 hostname
    2. 安装 docker
    3. 安装基础工具 kubeadm kubelet
    4. 关闭 swap
    5. 关闭防火墙如 firewalld
    6. 开启所有worker及master的路由模式
      • echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf && sysctl -p
    7. 同步时间 ntp 设置时区
      • yum install ntp && systemctl start ntpd.service
      • timedatectl set-timezone Asia/Shanghai
  2. 拉 k8s 依赖镜像
  3. 修改 kubelet cgroup-driver
  4. kubeadm init 这里需要确定具体的参数
    1. helm 安装 calico
  5. helm 安装 dashboard

部署

(Recommended) If you have plans to upgrade this single control-plane kubeadm cluster to high availability you should specify the –control-plane-endpoint to set the shared endpoint for all control-plane nodes.


建议:如果有计划为可用性升级该单节点部署,应该通过参数 --control-plane-endpoint 指定端点给控制平面

国内部署环境问题

镜像站

https://feisky.gitbooks.io/kubernetes/content/appendix/mirrors.html

1
2
3
4
5
6
7
# GCR(Google Container Registry)镜像
for i in `kubeadm config images list`; do
imageName=${i#k8s.gcr.io/}
docker pull registry.aliyuncs.com/google_containers/$imageName
docker tag registry.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.aliyuncs.com/google_containers/$imageName
done;
1
2
3
4
5
6
for i in `kubeadm config images list`; do 
imageName=${i#k8s.gcr.io/}
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done;
1
2
docker pull coredns/coredns:1.8.4
docker tag docker.io/coredns/coredns:1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4

端口开放列表

6443
30000-32767

基础工具安装 kubeadm kubelet

Letting iptables see bridged traffic
Make sure that the br_netfilter module is loaded. This can be done by running lsmod | grep br_netfilter. To load it explicitly call sudo modprobe br_netfilter.

As a requirement for your Linux Node’s iptables to correctly see bridged traffic, you should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config, e.g.

1
2
3
4
5
6
7
8
9
10
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

kubeadm

kubeadm is network provider-agnostic
kubeadm 网络组件不定提供

kubeadm.yaml

kubelet

会在 join 后才能正常启动

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally.

docker info |grep Cgroup
vi /etc/sysconfig/kubelet
修改 KUBELET_EXTRA_ARGS=–cgroup-driver=cgroupfs
systemctl show –property=Environment kubelet |cat

systemctl enable –now kubelet

处理启动失败的情况
1
journalctl -xefu kubelet

平台组件

1
2
3
4
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

网络组件

使每一个 pod 都有全集群唯一的虚拟IP地址

coredns (or kube-dns) should be in the Pending state until you have deployed the network add-on.


  • 实现方式
    • overlay network 覆盖性网络
    • non-overlay 非覆盖性
  • BGP 边界网关协议

  • Flannel
    • 被公认为是最简单的选择
  • Calico
    • BGP boarder gateway protocol 边界网关协议
    • 以其性能、灵活性而闻名
    • 在大规模集群中可以通过额外的 BGP route reflector 来完成
    • 基于 iptables 还提供了丰富的网络策略
    • 实现了 Kubernetes 的 Network Policy 策略(提供容器间网络可达性限制的功能)
  • OVS
    • Open VSwitch
    • 相对 calico 性价比比较高
    • 相对 calico 稳定性更好
Falnnel

https://github.com/flannel-io/flannel#deploying-flannel-manually

1
2
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml


  • 镜像国内服务器拉不下来,需要处理镜像地址
    • 上传到 dockerhub
    • 上传到阿里云镜像仓库(目前免费)
1
2
3
4
5
docker pull quay.io/coreos/flannel:v0.14.0
docker tag quay.io/coreos/flannel:v0.14.0 registry.cn-chengdu.aliyuncs.com/haowei_ch/flannel:v0.14.0
docker push registry.cn-chengdu.aliyuncs.com/haowei_ch/flannel:v0.14.0

# 自行替换文件里的镜像地址

1
2
3

kubectl apply -f kube-flannel.yml
kubectl get pods -n kube-system |grep flannel
kube-flannel.yml
1

Calico

Calico is a networking and network policy provider.
Calico supports a flexible set of networking options so you can choose the most efficient option for your situation, including non-overlay and overlay networks, with or without BGP.
Calico uses the same engine to enforce network policy for hosts, pods, and (if using Istio & Envoy) applications at the service mesh layer

https://docs.projectcalico.org/latest/introduction/
https://docs.projectcalico.org/getting-started/kubernetes/quickstart

**一定要修改 node ip detection method **

1
kubectl edit configmap/coredns -n kube-system
卸载
1
2
3
4
5
kubectl delete --grace-period=0 --force -f https://docs.projectcalico.org/manifests/tigera-operator.yaml

rm -f /etc/cni/net.d/10-calico.conflist
rm -f /etc/cni/net.d/calico-kubeconfig

通过 helm 安装

https://docs.projectcalico.org/getting-started/kubernetes/helm

1
2
3
4
5
6
7
helm repo add projectcalico https://docs.projectcalico.org/charts

helm show values projectcalico/tigera-operator --version v3.20.1

helm install calico projectcalico/tigera-operator --version v3.20.1

watch kubectl get pods -n calico-system

Dashboard 控制台

https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

1
2
3
4
5
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

# 修改相关信息

kubectl apply -f kube-dashboard.yml
1
2
# 查看容器状态
kubectl get pods -n kubernetes-dashboard

删除
切换 ns 到 kubernetes-dashboard
k delete $(k get pod -o name | grep dashboard)
k delete deploy dashboard-metrics-scraper kubernetes-dashboard

使用

Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19; use the “seccompProfile” field instead

1
2
kubectl proxy
http://127.0.0.1:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
helm 安装
1
2
helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm install dashboard kubernetes-dashboard/kubernetes-dashboard

kubectl proxy

open url

http://127.0.0.1:8001/api/v1/namespaces/default/services/https:dashboard-kubernetes-dashboard:443/proxy/#/login

Descheduler 重平衡工具

  • 已实现的调度策略
    • RemoveDuplicates 移除重复 pod
    • LowNodeUtilization 节点低度使用
    • RemovePodsViolatingInterPodAntiAffinity 移除违反pod反亲和性的 pod
    • RemovePodsViolatingNodeAffinity

多集群联邦

日志聚合

https://kubernetes.io/zh/docs/concepts/cluster-administration/logging/

目前的解决方案

filebreat => logstash => graylog => elasticsearch

rotate

需要 rotate 日志

k8s 默认脚本部署的话,存在一个 logrotate,每小时运行一次。 kubeadm 需要手动设置容器运行时来自动地轮转应用日志。

https://www.elastic.co/guide/en/beats/filebeat/7.14/_log_rotation.html

维护

https://kubernetes.io/docs/tasks/administer-cluster/

节点动态加入

1
2
3
4
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 139.9.33.44:6443 --token n6t4er.xtc9vaop9evvdqca \
--discovery-token-ca-cert-hash sha256:2a2c106c94414a23a43f8cd4a4c96120c91792c0837a90f36655f8f56e96aff4

CLI 工具

kubectx kubens

kubectx + kubens: Power tools for kubectl

命令行切换 ctx 或 namespace
上下文(集群) 或 命名空间

证书管理

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

1
kubeadm certs check-expiration

升级集群

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

配置修改

1
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' > kubeadm.cfg.yaml

添加证书许可DNS

https://blog.scottlowe.org/2019/07/30/adding-a-name-to-kubernetes-api-server-certificate/

修改 kubeadm.cfg.yaml

1
2
3
4
apiServer:
certSANs:
- "139.9.33.44"
- "127.0.0.1"

移动现有 cert (kubeadm 不支持覆盖)
mv /etc/kubernetes/pki/apiserver.{crt,key} .

重新生成
kubeadm init phase certs apiserver --config kubeadm.cfg.yaml

强制重启 api server
docker ps | grep kube-apiserver | grep -v pause

docker stop {ID}

使用

合并 kube config

1
2
3
4
5
6
KUBECONFIG=file1:file2:file3 kubectl config view --merge --flatten > out.txt
KUBECONFIG=config:zbanx.local kubectl config view --merge --flatten > config

# 拆分
KUBECONFIG=in.txt kubectl config view \
--minify --flatten --context=context-1 > out.txt