部署环境

系统版本 centos7.9
cpu架构 x86
节点：
k8s-master1 10.100.234.48
k8s-node1 10.100.234.41
k8s-node2 10.100.234.27

服务器环境配置

三台机器增加hosts配置
vi /etc/hosts

10.100.234.48 k8s-master1
10.100.234.41 k8s-node1
10.100.234.27 k8s-node2

三台虚拟机安装chrony配置时间同步

在主节点安装集群依赖环境：

yum install -y socat
yum install -y conntrack
yum install -y ebtables
yum install -y ipset
yum install -y ipvsadm

image.png

使用kubekey部署kubesphere和k8s

#创建目录
mkdir kubekey && cd kubekey
#下载kubekey
curl -sfL https://get-kk.kubesphere.io | sh -
#创建配置文件
./kk create config --with-kubernetes v1.23.10 --with-kubesphere v3.3.0

修改配置文件
vi config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: k8s-master1, address: 10.100.234.48, internalAddress: 10.100.234.48, user: root, password: "xx"}
  - {name: k8s-node1, address: 10.100.234.41, internalAddress: 10.100.234.41, user: root, password: "xx"}
  - {name: k8s-node2, address: 10.100.234.27, internalAddress: 10.100.234.27, user: root, password: "xx"}
  roleGroups:
    etcd:
    - k8s-master1
    control-plane:
    - k8s-master1
    worker:
    - k8s-node1
    - k8s-node2
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.23.10
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: docker
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    privateRegistry: ""
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []
  addons: []
...

修改hosts配置和对应的密码，以及roleGroups下的节点hostname。

国内网络环境下配置环境变量，执行下面命令：

export KKZONE=cn

创建集群

./kk create cluster -f config-sample.yaml

需要等待一段时间进行下载和安装
image.png

如果你需要通过域名访问kubesphere集群，需要配置nginx:

server {
        listen       80 ;
        server_name  docsk8s.xx.com;

         location / {
                proxy_pass http://10.100.234.48:30880/;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $https;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
                proxy_set_header WL-Proxy-Client-IP $remote_addr;
                proxy_redirect default;
                proxy_redirect http:// https://;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "$connection_upgrade";
        }
    }

http模块增加配置

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

将域名docsk8s.xx.com指向这里的nginx ip:port

打开 http://10.100.234.48:30880 访问kubesphere网站，或者通过配置域名访问。

创建springboot服务并打包镜像，上传到maven私服用于服务部署。

创建springboot服务，增加两个接口：

@RestController
@RequestMapping()
public class ApiController {

    @GetMapping("/api/test")
    public String test(){
		return "ok";
    }
    @GetMapping("/")
    public String check(){
        return "success";
    }
}

pom.xml增加docker plugin和私服配置

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>springbootdocker</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>SpringBootDockerExample</name>
    <description>Docker and Spring boot</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.3.1.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <docker.image.prefix>springio</docker.image.prefix>
        <docker.image.prefix>docker.xx.com</docker.image.prefix>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>com.spotify</groupId>
                <artifactId>dockerfile-maven-plugin</artifactId>
                <version>1.4.13</version>
                <executions>
                    <execution>
                        <id>default</id>
                        <goals>
                            <goal>build</goal>
                            <goal>push</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <repository>${docker.image.prefix}/${project.artifactId}</repository>
                    <tag>${project.version}</tag>
                    <buildArgs>
                        <JAR_FILE>${project.build.finalName}.jar</JAR_FILE>
                    </buildArgs>
                </configuration>
            </plugin>
        </plugins>
    </build>


</project>

根目录创建Dockerfile文件

FROM openjdk:11-jdk
VOLUME /tmp
ARG JAR_FILE
COPY target/${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

构建并push镜像

mvn clean package  dockerfile:build  dockerfile:push

生成镜像：
docker.xx.com/springbootdocker:0.0.1-SNAPSHOT

在kubesphere后台部署docker项目

平台管理-集群-项目-创建
image.png

平台管理-集群-配置-保密字典-创建
增加dockerhub镜像仓库
image.png

平台管理-集群-应用负载-工作负载-部署-创建
选择上述创建的项目，选择maven私服，创建容器
image.png
查看部署面板
image.png

集群内部访问：
1.通过clusterip访问
进入服务器，通过容器组IP地址访问服务
curl 10.233.81.23:8080/
curl 10.233.76.27:8080/
image.png
2.直接访问服务名，依靠 DNS 解析
多个服务内部访问：

curl springboot-deployment:8080

相同项目中通过 service:port 访问，不同项目中通过 service.namespace:port 访问。

image.png

配置网关和路由，在集群外访问服务

平台管理-集群-集群设置-网关设置-启用网关

image.png
平台管理-集群-项目-刚才创建的项目-网关
查看项目网关
image.png

平台管理-集群-应用负载-工作负载-服务-创建
选择刚才创建的项目和对应的部署，端口填写服务8080端口。
image.png
查看详情
image.png

平台管理-集群-应用负载-应用路由-创建
image.png

访问服务
image.png

在集群外的其他机器配置hosts，IP地址为网关IP地址
10.100.234.48 springbootdocker.com

访问服务接口：
curl http://springbootdocker.com:32263/
image.png

更多

官方文档：

https://github.com/kubesphere/kubekey/blob/master/README_zh-CN.md

启用devops:

https://www.kubesphere.io/zh/docs/v3.3/pluggable-components/devops/

arm环境安装注意事项：
  1. config-sample.yaml中配置的hosts需要增加arch配置
    如
    {name: centos1, address: 10.211.55.5, internalAddress: 10.211.55.5, user: root, password: "123456", arch: arm64}
 # If you install Kubernetes on ARM, add "arch: arm64". For example, {...user: ubuntu, password: Qcloud@123, arch: arm64}.
  1. etcd提示报错export ETCD_UNSUPPORTED_ARCH=arm64
    其实已经在终端执行了此命令，仍然提示报错。
    查看报错日志
    tail -f /var/log/messages
    vi /etc/etcd.env
    增加一行
    ETCD_UNSUPPORTED_ARCH=arm64
    重启
    systemctl restart etcd
    重新执行命令安装即可