rancher + drone + nexus registry + gitea 自动化部署指南

Sep 1, 2023 · 5 min read ·

Share on:

Overview

[TOC]

部署架构

K3s：轻量级k8s集群，没有那么注重安全，针对LOT环境特殊优化。
RKe：另外一种轻量级k8s集群，相对K3s更注重安全，暂时没有研究。
Rancher：一种K8s集群管理工具，当然也可以用来管理K3s、RKe集群。
部署结构

搭建环境

前置准备

每一个VM节点，请务必保证hostname、IP是固定且唯一的，不能重复或者冲突；同时设置域名neuxs.hkyx.com的解析地址。

 1# 设置hostname
 2$ sudo hostnamectl set-hostname xxxx
 3
 4$ sudo vi /etc/hosts
 5127.0.1.1       xxxx
 6
 7# 设置静态IP
 8$ sudo vi /etc/network/interfaces
 9iface ens18 inet static
10address 192.168.10.xxx
11
12# 设置域名解析，保证每个VM节点都能识别该域名
13$ sudo vi /etc/hosts
14nexus.hkyx.com	192.168.10.253

请保证可以访问外网（师夷长技以制夷）
每个VM节点的用户都需要加入sudo组，可以执行sudo命令

每一个VM节点都需要关闭swap分区

1sudo swapoff -a
2sudo sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab

证书

因为nexus作为镜像仓库要用到nexus.hkyx.com的域名，且rancher访问域名时必须使用https协议，所以我们需要生产https证书，并且同步ca证书到每一个VM节点。

生产证书

登陆cicd-node，生成sans自签名证书：

 1# 创建ssl目录
 2$ cd ~ && mkdir ssl-sans && cd ssl-sans
 3
 4# 生成根证书密钥ca.key
 5$ openssl genrsa -out ca.key 4096
 6
 7# 生成自签名根证书ca.crt
 8$ openssl req -x509 -new -nodes -sha512 -days 3650 \
 9 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=nexus.hkyx.com" \
10 -key ca.key \
11 -out ca.crt
12 
13# 生成nexus.hkyx.com.key
14$ openssl genrsa -out nexus.hkyx.com.key 4096
15 
16# 生成nexus.hkyx.com.csr
17$ openssl req -sha512 -new \
18    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=nexus.hkyx.com" \
19    -key nexus.hkyx.com.key \
20    -out nexus.hkyx.com.csr
21
22# 生成v3.ext,sans扩展
23$ cat > v3.ext <<-EOF
24authorityKeyIdentifier=keyid,issuer
25basicConstraints=CA:FALSE
26keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
27extendedKeyUsage = serverAuth
28subjectAltName = @alt_names
29
30[alt_names]
31DNS.1=nexus.hkyx.com
32DNS.2=nexus.hkyx
33DNS.3=hostname
34EOF
35
36# 生成nexus.hkyx.com.crt
37$ openssl x509 -req -sha512 -days 3650 \
38  -extfile v3.ext \
39  -CA ca.crt -CAkey ca.key -CAcreateserial \
40  -in nexus.hkyx.com.csr \
41  -out nexus.hkyx.com.crt

同步证书

登陆cicd-node节点，复制nexus.hkyx.com.crt证书到每一个节点：

1# 复制到cicd-node节点自身
2$ sudo cp ~/ssl-sans/*.crt /tmp
3
4# 复制到远程节点: rancher server01 worker01 worker02
5$ sudo scp ~/ssl-sans/*.crt mac@192.168.10.203:/tmp
6$ sudo scp ~/ssl-sans/*.crt mac@192.168.10.204:/tmp
7$ sudo scp ~/ssl-sans/*.crt mac@192.168.10.205:/tmp
8$ sudo scp ~/ssl-sans/*.crt mac@192.168.10.206:/tmp

安装证书

在所有节点（cicd-node,rancher,server01,worker01,worker02）中分别执行如下命令：

1$ sudo apt-get install ca-certificates
2# 移动证书到/usr/share/ca-certificates/local
3$ sudo mkdir -p /usr/share/ca-certificates/local 
4$ sudo mv /tmp/*.crt /usr/share/ca-certificates/local
5# 安装证书，敲空格选中nexus.hkyx.com.crt
6$ sudo dpkg-reconfigure ca-certificates

设置cicd-node节点

安装Docker19.03 + Docker-Compose1.25.4

 1$ curl https://releases.rancher.com/install-docker/19.03.sh | sh
 2
 3$ sudo vi /etc/docker/daemon.json
 4{
 5  "registry-mirrors": ["http://hub-mirror.c.163.com"],
 6  "insecure-registries":["192.168.10.253","nexus.hkyx.com"]
 7}
 8
 9$ sudo systemctl restart docker
10
11$ curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
12
13$ sudo chmod +x /usr/local/bin/docker-compose
14
15$ sudo docker-compose --version

安装gitea

 1$ mkdir ~/gitea && cd ~/gitea
 2$ sudo vi docker-compose.yml
 3version: "3"
 4 
 5networks:
 6  gitea:
 7    external: false
 8 
 9services:
10  server:
11    image: gitea/gitea:1.16.7
12    container_name: gitea
13    environment:
14      - USER_UID=1000
15      - USER_GID=1000
16    restart: always
17    networks:
18      - gitea
19    volumes:
20      - /home/data:/data  # /home/data可以替换成你想要的挂载目录
21      - /etc/timezone:/etc/timezone:ro
22      - /etc/localtime:/etc/localtime:ro
23    ports:
24      - "3000:3000" # 3030可以替换成你想要的端口
25      - "2222:22" # 322可以替换成22
26$ sudo docker-compose up -d

安装drone

 1$ mkdir ~/drone && cd ~/drone
 2$ sudo vi docker-compose.yml
 3version: '3'
 4services:
 5  drone-server:
 6    restart: always
 7    image: drone/drone:2
 8    ports:
 9      - "1080:80"
10    volumes:
11      - /var/drone:/var/lib/drone/
12      - /var/drone_data:/data/
13    environment:
14      - DRONE_GITEA_SERVER=http://192.168.10.253:3000
15      - DRONE_GITEA_CLIENT_ID=581c2741-a131-40e3-bd16-1197a9a2b06b
16      - DRONE_GITEA_CLIENT_SECRET=6yiusBvtBkIcltm2iIQ4siEgylpp7WN50Ld5pjE0PhE8
17      - DRONE_SERVER_HOST=192.168.10.253:1080
18      - DRONE_SERVER_PROTO=http
19      - DRONE_RPC_SECRET=34e6c77e95ea65ac20d305fad7980b09
20      - DRONE_GIT_ALWAYS_AUTH=true
21      - DRONE_GIT_USERNAME=tfnick
22      - DRONE_GIT_PASSWORD={你的密码}
23      - DRONE_USER_CREATE=username:tfnick,admin:true
24  drone-runner-docker:
25    restart: always
26    image: drone/drone-runner-docker:1
27    ports:
28      - "3001:3000"
29    volumes:
30      - /var/run/docker.sock:/var/run/docker.sock
31    environment:
32      - DRONE_RPC_PROTO=http
33      - DRONE_RPC_HOST=drone-server
34      - DRONE_RPC_SECRET=34e6c77e95ea65ac20d305fad7980b09
35      - DRONE_RUNNER_NAME=drone-runner-docker
36      - DRONE_RUNNER_CAPACITY=2
37      
38$ sudo docker-compose up -d

备注：

DRONE_GITEA_CLIENT_ID: gitea中配置oath2的client_id

DRONE_GITEA_CLIENT_SECRET: gitea中配置的oath2的client secret

安装nexus + nginx

 1$ sudo mkdir -p /var/nexus-data && sudo chmod 777 -R /var/nexus-data
 2$ sudo docker run --restart=always -tid -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 8084:8084 --name nexus -e NEXUS_CONTEXT=nexus -v /var/nexus-data:/nexus-data  docker.io/sonatype/nexus3 
 3
 4$ sudo mkdir -p /var/docker-nginx/nginx && sudo mkdir -p /var/docker-nginx/logs
 5
 6# 复制证书到宿主机的/var/docker-nginx/nginx/
 7$ sudo cp -r ~/ssl-sans/ /var/docker-nginx/nginx/
 8
 9$ sudo docker run -p 80:80 -p 443:443 --name nginx --restart=always -v /var/docker-nginx/nginx:/etc/nginx -v /var/docker-nginx/logs:/var/log/nginx -d nginx
10
11# 配置/var/docker-nginx/nginx/conf.d/default.conf
12$ sudo vi /var/docker-nginx/nginx/conf.d/default.conf
13upstream nexus_website {
14    server 192.168.10.253:8081;
15}
16upstream nexus_docker_hosted {
17    server 192.168.10.253:8082;
18}
19upstream nexus_docker_group {
20    server 192.168.10.253:8084;
21}
22
23server {
24    listen 80;
25    
26    # nexus对外的域名 
27    server_name nexus.hkyx.com;
28
29    #https配置开始
30    listen       443 ssl;
31    #这里是nginx的证书配置
32    ssl_certificate      /etc/nginx/ssl-sans/nexus.hkyx.com.crt;
33    ssl_certificate_key  /etc/nginx/ssl-sans/nexus.hkyx.com.key;
34
35    ssl_session_cache    shared:SSL:1m;
36    ssl_session_timeout  5m;
37
38    ssl_ciphers  HIGH:!aNULL:!MD5;
39    ssl_prefer_server_ciphers  on;
40    #https配置结束
41
42    access_log /var/log/nginx/nexus.hkyx.com.log main;
43        
44    # disable any limits to avoid HTTP 413 for large image uploads
45    client_max_body_size 0;
46    # required to avoid HTTP 411: 
47    chunked_transfer_encoding on;
48    # 设置默认使用推送代理
49    set $upstream "nexus_docker_hosted";
50    # 当请求是GET，也就是拉取镜像的时候，这里改为拉取代理，如此便解决了拉取和推送的端口统一
51    if ( $request_method ~* 'GET') {
52        set $upstream "nexus_docker_group";
53    }
54    # 只有本地仓库才支持搜索，所以将搜索请求转发到本地仓库，否则出现500报错
55    if ($request_uri ~ '/search') {
56        set $upstream "nexus_docker_hosted"; 
57    }  
58    index index.html index.htm index.php;
59    location / {
60            proxy_pass http://$upstream;
61            proxy_set_header Host $host;
62            proxy_connect_timeout 3600;
63            proxy_send_timeout 3600;
64            proxy_read_timeout 3600;
65            proxy_set_header X-Real-IP $remote_addr;
66            proxy_buffering off;
67            proxy_request_buffering off;
68            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
69            proxy_set_header X-Forwarded-Proto http;
70    }
71}
72
73server {
74    listen 80;
75    server_name nexus-web.hkyx.com;
76    access_log /var/log/nginx/nexus-web.log main;
77    index index.html index.htm index.php;
78    location /nexus { 
79            proxy_pass http://nexus_website/nexus;
80            proxy_set_header Host $host;
81            client_max_body_size 512m;
82            proxy_connect_timeout 3600;
83            proxy_send_timeout 3600;
84            proxy_read_timeout 3600;
85            proxy_buffering off;
86            proxy_request_buffering off;
87    }
88}

设置

配置tf-docker-hosted仓库，开启http端口8082
配置tf-docker-proxy仓库，开启http端口8083
- Remote storage: https://mirror.baidubce.com
- Docker Index: Use Docker Hub
配置tf-docker-group仓库，开启http端口8084
- Members: 选中 tf-docker-hosted,tf-docker-proxy
Security - Realms，激活Docker Bearer Token Realm，以便后续其他节点可以执行sudo docker login 命令

访问

https://nexus.hkyx.com

设置rancher节点

安装Docker19.03

1$ curl https://releases.rancher.com/install-docker/19.03.sh | sh
2
3$ sudo vi /etc/docker/daemon.json
4{
5  "registry-mirrors": ["http://hub-mirror.c.163.com"],
6  "insecure-registries":["192.168.10.253","nexus.hkyx.com"]
7}
8
9$ sudo systemctl restart docker

安装Rancher2.7.6

1$ sudo mkdir -p /var/rancher-data
2$ sudo docker run -d --privileged --restart=unless-stopped -p 80:80 -p 443:443 -v /var/rancher-data:/var/lib/rancher/  -v /usr/share/ca-certificates/local:/container/certs -e SSL_CERT_DIR="/container/certs" --add-host nexus.hkyx.com:192.168.10.253 --add-host nexus-web.hkyx.com:192.168.10.253 --privileged  rancher/rancher:v2.7.6

配置Rancher2.7.6

创建集群k3s-cluster
- 进入rancher首页，依次点击创建-自定义，输入集群名称后，点击 创建进行保存。
- 查看用于注册k3s集群server01节点的命令
- 查看用户注册k3s集群worker01,worker02节点的命令
创建集群的secret，后续从nexus拉取镜像会用到。（重要）

进入k3s-cluster集群，依次点击存储-Secret-创建-镜像仓库，填入nexus-registry的地址和凭证信息。

设置server01节点

注册该节点

1$ curl --insecure -fL https://192.168.10.203/system-agent-install.sh | sudo  sh -s - --server https://192.168.10.203 --label 'cattle.io/os=linux' --token v98g582t55jm7vld5jc8pdmkw49dpfxwlgk55qqshsgtccwlrpzqkr --ca-checksum b94ba414a26126769787e01286bd861a15f6991c1dc221bbd7e13ab33e5f0cd5 --etcd --controlplane --worker

设置worker01节点

注册该节点

1$ curl --insecure -fL https://192.168.10.203/system-agent-install.sh | sudo  sh -s - --server https://192.168.10.203 --label 'cattle.io/os=linux' --token v98g582t55jm7vld5jc8pdmkw49dpfxwlgk55qqshsgtccwlrpzqkr --ca-checksum b94ba414a26126769787e01286bd861a15f6991c1dc221bbd7e13ab33e5f0cd5 --worker

设置worker02节点

注册该节点

1$ curl --insecure -fL https://192.168.10.203/system-agent-install.sh | sudo  sh -s - --server https://192.168.10.203 --label 'cattle.io/os=linux' --token v98g582t55jm7vld5jc8pdmkw49dpfxwlgk55qqshsgtccwlrpzqkr --ca-checksum b94ba414a26126769787e01286bd861a15f6991c1dc221bbd7e13ab33e5f0cd5 --worker

节点注册问题排查

问题：error applying plan -- check rancher-system-agent.service logs

排查：sudo journalctl -eu rancher-system-agent -f

操作：

rancher中删除节点
login as root user, run command: rm -fr /var/lib/rancher/*
重新注册节点

打通gitea+drone+nexus+rancher

待续

参考

rancher + drone ci/cd