From 3a1924f5ed4fe70f2cad42067cad67f16d6b0f76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=B8=80=E5=8F=B6=E4=B9=8B=E7=A7=8B?= Date: Tue, 26 Jan 2021 20:15:53 +0800 Subject: [PATCH] init --- DOCUMENTS/build-a-docker-image.md | 83 ++++ ...g-certificates-for-jenkins-kubernetes-plugin.md | 47 ++ DOCUMENTS/container-guide.md | 548 +++++++++++++++++++++ DOCUMENTS/how-to-get-kubectl-bin.md | 78 +++ DOCUMENTS/jenkins-pipline-usage.md | 163 ++++++ DOCUMENTS/mapping-external-services.md | 149 ++++++ README.md | 31 +- 7 files changed, 1097 insertions(+), 2 deletions(-) create mode 100644 DOCUMENTS/build-a-docker-image.md create mode 100644 DOCUMENTS/configuring-certificates-for-jenkins-kubernetes-plugin.md create mode 100644 DOCUMENTS/container-guide.md create mode 100644 DOCUMENTS/how-to-get-kubectl-bin.md create mode 100644 DOCUMENTS/jenkins-pipline-usage.md create mode 100644 DOCUMENTS/mapping-external-services.md diff --git a/DOCUMENTS/build-a-docker-image.md b/DOCUMENTS/build-a-docker-image.md new file mode 100644 index 0000000..6878483 --- /dev/null +++ b/DOCUMENTS/build-a-docker-image.md @@ -0,0 +1,83 @@ +build-a-docker-image.md +----------------------- +``` +@version 180911:2 +@author zhangxuhong +``` + +Name +---- + + build-a-docker-image - 构建Docker镜像. + + +Table of Contents +----------------- + +* [Name](#name) +* [Image Manifest Management 镜像依赖管理](#image-manifest-management) +* [Minimize Docker Image 缩减 Docker 镜像大小](#minimize-docker-image) + + +Image Manifest Management 镜像依赖管理 +-------------------------------------- + +### 构建普通镜像 + +目前harbor中的镜像库包含以下几个项目 + +- lib, 公网上的镜像备份 +- infrastructure, 私有基础镜像 +- test, 测试用, 随便用, test发版用镜像 +- beta, beta发版用镜像 +- prod, prod发版用镜像 + +比如现在正在构建项目: + +- harbor02.juejin.id/prod/demo-api + +该项目依赖: + +- harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module:0.0.3 +- harbor02.juejin.id/lib/framework:0.12.3 + +那么在引用依赖和发版的时候, 都应该使用latest tag的镜像, 这样可以保证每次新发布的时候, 可以获取最新的更新(需要手动把本地镜像缓存清理一下). +所以依赖应该是: + +- harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module:latest +- harbor02.juejin.id/lib/framework:latest + +这样当这两个项目有bugfix的时候, 只需要重新构建你的项目, 而不必修改Dockerfile. + +发布的时候, jenkins会自动执行你指定的jenkinsfile来把构建完毕的镜像推送到harbor, 镜像tag通常是git提交的hash值(这是在jenkinsfile自动生成的,不必手动修改). + +- harbor02.juejin.id/prod/demo-api:xxxxx 用来让CI程序进行发版 + + +### 构建基础镜像 + +制作基础镜像也是同样的流程, 比如 harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module 修正了一个bug, 那么版本变为: + +- harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module:0.0.4 + +同时应再推送一个 + +- harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module:latest + +版本方便其他依赖该镜像的项目用来更新. + + +Minimize Docker Image 缩减 Docker 镜像大小 +------------------------------------------ + +可以参考这篇文档: + +- [https://www.sandtable.com/reduce-docker-image-sizes-using-alpine/](https://www.sandtable.com/reduce-docker-image-sizes-using-alpine/) + +基本思路就是用Alpine, 然后把不用的都删掉. 懒得删就用多端构建, 只把需要的bin和so拷贝过去. + +有个实用的工具 dlayer, 可以分析每一行dockerfile命令带来的镜像膨胀. + +- [https://github.com/orisano/dlayer](https://github.com/orisano/dlayer) + +已经在docker-client01v.lobj.juejin.id部署, 可以直接使用. diff --git a/DOCUMENTS/configuring-certificates-for-jenkins-kubernetes-plugin.md b/DOCUMENTS/configuring-certificates-for-jenkins-kubernetes-plugin.md new file mode 100644 index 0000000..7b8aca0 --- /dev/null +++ b/DOCUMENTS/configuring-certificates-for-jenkins-kubernetes-plugin.md @@ -0,0 +1,47 @@ +configuring-certificates-for-jenkins-kubernetes-plugin.md +--------------------------------------------------------- +``` +@version 180807:1 +@author zhangxuhong +``` + +Name +---- + + configuring-certificates-for-jenkins-kubernetes-plugin - 配置jenkins kubernetes查件的认证. + + +Table of Contents +----------------- + +* [Name](#name) +* [Reference 参考文档](#reference) + + + +Reference 参考文档 +------------------ + +* [简单来讲这个文章可以搞定一切](https://illya-chekrygin.com/2017/08/26/configuring-certificates-for-jenkins-kubernetes-plugin-0-12/) + + +Generate Kubernetes server certificate key 生成 kubernetes 认证 +--------------------------------------------------------------- + +首先 kubectl 的配置文件默认在 ```~/.kube/config```, 我们需要 decode 其中的 certificate-authority-data 来获得ca证书. + +echo {PASTE certificate-authority-data HERE} | base64 -d > ca.crt + +将生成的 ca.crt 粘贴到 Kubernetes server certificate key 中. + +同样步骤将 client-certificate-data 生成 client.crt, client-key-data 生成 client.key. + +最后用 openssl 生成 PKCS12 格式的客户端认证. + +openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt + +注意生成需要输入 Export Password 这是必须的, 空密码会造成后续配置产生问题. + +然后将生成的 cert.pfx 上传, 并且填写密码. (注意这里有bug, 直到你填写完毕密码之前, 他都提示你未上传. 是没有上传成功提示的.) + +最后可以点击Test Connection 按钮来测试配置是否正确. Connection test successful 即为配置正确. \ No newline at end of file diff --git a/DOCUMENTS/container-guide.md b/DOCUMENTS/container-guide.md new file mode 100644 index 0000000..f97c729 --- /dev/null +++ b/DOCUMENTS/container-guide.md @@ -0,0 +1,548 @@ +container-guide.md +------------------ +``` +@version 180521:1 +@author zhangxuhong +``` + +Name +---- + + container-guide - 容器化参考文档. + + +Table of Contents +----------------- + +* [Name](#name) +* [Prepare For Container](#prepare-for-container) + + +Prepare For Container +--------------------- + +- 开始 + +首先, 请阅读这个gitbook来补充有关kubernetes的相关知识. [https://jimmysong.io/kubernetes-handbook/](https://jimmysong.io/kubernetes-handbook/) + + - 简单了解 kubernetes 架构 + +- 需求 + +我们假设接到了个计数器的需求 access-counter, 该需求要求用户传入自己的suid, 然后在redis中对该suid进行加一操作. +输出 json 结构为: {"suid":"ZFnUF6YraFRqRbY7izMm", "count":12}. +如果用户传入的suid为空, 则调用 kubernetes 集群中的 suid-generator 接口生成一个suid, 然后按照上面的格式返回. +注意: 本示例只是为了展示 kubernetes 使用, 这个例子存在很明显的问题, 比如没有鉴权, 以及生成 suid 不应该由计数器负责. + +- 改造 + +我们的容器化方案是 docker + kubernetes, 因此我们的第一个步骤就是将我们现有的业务装到docker中. +在装入 docker 之前, 我们需要简单修改一下程序来适应容器环境的一些需求. + +- log 问题 + + 由于我们默认容器是不映射实体存储设备的, 也就意味着我们的容器销毁后里面的内容就全部丢失了, 所以在容器内部写日志本身就毫无意义. + 因此, 我们需要将 info 级别的日志直接打印到 stdout, error 级别的日志直接打印到 stderr, 然后通过特定的日志收集程序进行统一处理. + + - 针对 PHP 的场景 + + ``` + // 打印到 stdout + error_log($log, 3, "php://stdout"); + + // 打印到 stderr + error_log($log, 3, "php://stderr"); + // 注意 error_log 函数不是二进制安全的, 意味着如果 $log 变量中的字符含有 "\0" 的话, log 会被截断, 后半部分会丢失. + // 要么过滤文本中的 \0, 要么对文本进行转义(不推荐, 会导致日志人肉不可读), 要么保证文本没有 \0 (比如日志是你自己写的字面量). + ``` + + - 针对 lua 的场景 + + ``` + -- 打印到 stdout + io.stdout.write(log) + + -- 打印到 stderr + io.stderr.write(log) + ``` + + - 针对 go 的场景 + + ``` + // 打印到 stdout + fmt.Fprintln(os.Stdout, log) + + // 打印到 stderr + fmt.Fprintln(os.Stderr, log) + os.Stderr.WriteString(log) + logInstance := log.New(os.Stderr, "", 0) + logInstance.Println(log) + + // 总之 go 想打印的话方式还是很多的. + ``` + + - 针对 nodejs 的场景 + + ``` + // @todo: 待好心的同学有时间补完这里, 我不会写js ... + ``` + + 注意以上只是你自己写的日志, 你的 runtime (例如: php-fpm, luajit, node.js) 本身也会报错, 你使用的框架也会报错. + 因此还需要根据场景将 runtime 和框架的错误日志也写到 stderr. 否则线上出了故障要看日志只能 attach 到容器上去翻看了. + 然后容器如果是触发故障就崩溃, kubernetes 会自动重启故障, 你的故障日志就消失不见了. + 至于系统日志, 则会由 systemctl 统一接管, 可以用 journalctl -u {systemctlUnitName} -f 查看, 不用担心. + +- 容器互相调用问题 + + 既然容器隔离开了, 那么容器间怎么通信呢? 其实很简单, 直接调用容器的 service name, kubernetes 的内置 dns 就可以解析了. + + - nginx 配置样例 + + ``` + location ~ \.php$ { + if ( $fastcgi_script_name ~ \..*\/.*php ) { + return 403; + } + include fastcgi.conf; + fastcgi_pass ac-counter:9000; + fastcgi_index index.php; + } + ``` +- php 代码调用集群内其他服务的问题 + + 同上, 直接写 service name 即可. + + - 例子 + + ``` + /** + * config here + */ + $conf = array( + 'global' => array( + 'global_id' => 'access-counter', + 'folder' => '/data/repo/access-counter/', + ), + 'log' => array( + 'open' => true, + 'address' => '/data/repo/access-counter/logs/', // the '/' at end of line is necessary + 'split' => 'day', // options: month, day, hour, minute + ), + 'cache' => array( + 'access_counter_cache' => array( + 'host' => 'ac-counter-rds', + // 'host' => '127.0.0.1', + 'port' => 6379, + 'pass' => null, + 'database' => 0, + 'timeout' => 0, + ), + ), + 'suid_generator_api' => 'http://suid-generator/v1/gen_suid?src=%', + // 'suid_generator_api' => 'http://suid-generator-api-ms.juejin.im/v1/gen_suid?src=%', + ); + ``` + +- php 代码调用集群外服务问题. + + 同样, 写 service name, 不过我们要建立一个代理用的 service, 我们在下面的小节讲述这个问题. + + +- 生产环境问题 + + 我们尽量遵循原则 "不在代码中内嵌环境信息" 的原则. 所以我们需要通过外部配置文件来根据环境来进行配置. + 我们来列一下我们需要根据生产环境来切换的资源 + - 接口 + - 内部接口 + 好说, 生产环境内的接口就应该是你想要的, 直接调用 service name. + - 外部接口 + 代理 service, 通过 jenkinsfile 来控制具体映射关系 + + - 数据库 + 同接口 + + - 逻辑 + 这个是最难的, 比如我们想在接口中输出当前环境是 test, beta 还是 prod. 这时候就必须让代码(逻辑)感知到环境. + 注意, 这种能没有就不要有. 他破坏了我们代码的可部署性, 试想一下哪天我们多了个环境叫beta2, 你没准就要痛苦的修改1000多个repo的代码. + 因此, 良好的设计是, 获取当前环境的名称, 然后打印出来, 这样逻辑只是"获取环境参数, 并打印"跟环境无关. + 针对 php-fpm 的场景, 我们有几种方式获取当前环境. + - 将环境写入nginx, 用fcgi-param获取 + - 将 www.conf 配置文件的 clear_env = no 取消注释, 添加环境变量例如 + ``` + env["SERVER_ENV"] = $SERVER_ENV + ``` + 然后 我们在 deployments 文件中设置环境变量 + ``` + # ac-counter-deployment.yaml + # + # @version 180806:2 + # @author zhangxuhong + + kind: Deployment + apiVersion: apps/v1 + metadata: + name: ac-counter + labels: + name: ac-counter + role: backend + pl: php + application: php + version: 7.2.9 + division: infrastructure + spec: + replicas: 3 + selector: + matchLabels: + name: ac-counter + strategy: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 25% + maxSurge: 25% + template: + metadata: + labels: + name: ac-counter + spec: + containers: + - name: ac-counter + image: __IMAGE__ + imagePullPolicy: Always + ports: + - name: ac-counter + containerPort: 9000 + protocol: TCP + env: + - name: SERVER_ENV + value: "test" + ``` + - 针对 php-cli 直接使用 getenv, 或者获取命令行参数都可以. + + +Load Repo Into Container +------------------------ + +- Dockerfile + +首先我们改造好repo后, 接下来就可以开始装入容器了. 下面开始编写Dockerfile. + +构建镜像的注意事项详见 [build-a-docker-image.md](./build-a-docker-image.md) + +- nginx docker file + +``` +# ac-counter-ngx.dockerfile +# Dockerfile for demo ac-counter +# This docker file base on harbor02.juejin.id/lib/php:7.2.9-fpm-alpine3.8 +# @version 180719:2 +# @author zhangxuhong +# + +# base info +FROM harbor02.juejin.id/infrastructure/nginx-1.14.0-centos:latest +MAINTAINER zhangxuhong +USER root + +# copy config to /data/apps/nginx/conf/vhost/ +COPY ./config/nginx/ /data/apps/nginx/conf/vhost/ + +# define health check +HEALTHCHECK --interval=5s --timeout=3s CMD curl -fs http://127.0.0.1:80/status?src=docker_health_check -H"Host:access-counter-api.juejin.im" || exit 1 + +# run php-fpm +EXPOSE 80 +ENTRYPOINT ["/data/apps/nginx/sbin/nginx", "-g", "daemon off;"] +``` + + +- access-counter docker file + +``` +# ac-counter.dockerfile +# Dockerfile for demo access-counter +# This docker file base on harbor02.juejin.id/lib/php:7.2.9-fpm-alpine3.8 +# @version 180719:2 +# @author zhangxuhong +# + +# base info +FROM harbor02.juejin.id/lib/php:7.2.9-fpm-alpine3.8 +MAINTAINER zhangxuhong +USER root + +# init extension +RUN apk add --update --no-cache --virtual .build-deps \ + curl \ + g++ \ + gcc \ + gnupg \ + libgcc \ + make \ + alpine-sdk \ + autoconf +RUN pecl install redis-4.1.1 && docker-php-ext-enable redis + +# copy repo to /data/repo +COPY . /data/repo/access-counter/ + +# define health check +HEALTHCHECK --interval=5s --timeout=3s CMD netstat -an | grep 9000 > /dev/null; if [ 0 != $? ]; then exit 1; fi; + +# run php-fpm +EXPOSE 9000 +ENTRYPOINT ["php-fpm"] +``` + +装入完毕后开始构建镜像准备本地测试. + +``` +docker build ./ -t suid-generator +``` + +然后运行镜像进行测试. + +``` +docker run suid-generator + +docker exec -i -t {docker id} /bin/sh +curl http://{docker-port-ip}/status?src=tester -H"Host: access-counter-api.juejin.im" +``` + +如果curl正常返回结果就代表测试成功了. + + +- 准备 kubernetes 配置文件 + + - Deployment + + Deployment文件负责描述整个部署的 Pods 和 ReplicaSets. + + - Service + + Service 负责映射配置, Service将服务名称与具体的Pod及暴露的端口映射到一起这个映射关系就叫endpoints. + Service 映射外部IP或者域名见 [mapping-external-services.md](./mapping-external-services.md) + + - Ingress + + Ingress 负责配置负载均衡, 根据提供的域名和path将业务路由到指定的service. + + - Endpoints + + endpoint 用来描述 service 对应的流量关系. + + - 具体的配置文件请结合 access-counter 项目学习. + + +Configure CI/CD +--------------- + +- Jenkinsfile + + Jenkinsfile 其实是Groovy脚本, 通过配置来描述部署过程和配置. + + 具体编写和注意事项见 [jenkins-pipline-usage.md](./jenkins-pipline-usage.md) + +- 创建 pipline jenkins 任务 + + 注意最好按照我们的命名规则 {repoName}.{clusterNamespace}.{clusterName} 来给jenkins 任务命名. + + - 选中 Build when a change is pushed to GitLab. 注意这段话后面的就是webhook地址 + - 点击 Advanced 按钮, 勾选下面的 Allowed branches, 选择 Filter branches by regex 然后填写^{yourBranchName}$ + - 点击下面的 Generate 按钮生成 webhook token + - Pipeline -> Definition -> Pipline Script from SCM + - SCM 选择 Git + - 填写git中的地址到 Repository URL. 注意这里有个坑, 需要把我们的gitlab服务器的域名换成IP, 至于为什么, 我弄了12小时也没弄清楚... + - Credentials 选择已经填写好的 jenkins01 + - Branches to build 修改成上面Build Triggers填写的一样的 \*/{yourBranchName} + - Script Path 填写 config/jenkins/{yourJenkinsFile} + - 最后点击左下角的 save + +- 配置 gitlab trigger + + - 进入到repo, 选择左侧的 Settings -> Integrations + - 填写上面得到的 webhook地址 和生成的 token. + - 点击 Add webhook. + +- 多生产环境问题 + + - 没错, 我们有test, beta, prod 三个环境, 因此每个repo你都要重复上面无聊的工作3次. (暂时还没想好解决方案) + + +Release ! +--------- + +- 立刻推送到你想要发版的分支出发 webhook 来体验一下 CI/CD 吧 + +Panic +----- + +- 如何debug? + - 容器内bug + - kubectl 直接查看日志 + - 接执行 kubectl logs {podName} --namespace={yourDeploymentNamespace} 查看日志, -f 参数可以监听日志. + - 定位机器进入node节点查看docker日志 + - 在 kubernetes master 执行 kubectl get pods --namespace={yourDeploymentNamespace} 查找当前 pods 位于哪台机器上 + - kubectl describe pod {podName} 查看 pod 所在机器(node). + - 在目标机器执行 docker ps -a 查看进程名称. + - 最后通过 docker logs {process_name} 查看进程日志. + - 进入容器debug + - 进入容器所在节点主机. + - 执行 docker exec -i -t {CONTAINER_ID} /bin/bash + - 集群bug + - 使用 journalctl -u {unit_name} -t 查看想查看的集群进城日志, 例如: docker, kubelet. + + +- 如何扩容? + + - 直接修改Deployment文件中的replica数量, 然后CI流程重新部署. + + - 用kubectl命令 + + ``` + kubectl autoscale deployment {deploymentName} --min=2 --max=10 + kubectl scale --replicas=3 -f {deploymentFile} + ``` + + - 建议除了测试以外用第一种进行扩容, 否则线上与git中的 deployment 文件不一致, 再次发办可能会面临风险. + + +Remove +------ + +- 如何删除? + + 我们部署了总计三个 resource: deployment, service, ingress. + 那么直接执行kubectl delete {resourceName} {repoName} --namespace={yourDeploymentNamespace} 即可. + 例如: + + ``` + kubectl delete deployment {repoName} --namesapce=test + kubectl delete service {repoName} --namesapce=test + kubectl delete ingress {repoName} --namesapce=test + ``` + +Tips & Reference +---------------- + +如果感兴趣可以阅读其他参考资料书籍(按推荐程度排序): + +- [Kubernetes Handbook](https://jimmysong.io/kubernetes-handbook/) +- Docker 容器与容器云(第2版) +- Cloud Native Go - 基于Go和React的web云原生应用构建指南 +- [Kubernetes官方文档](https://kubernetes.io/docs/home/?path=users&persona=app-developer&level=foundational) +- [Traefik 官方文档](https://docs.traefik.io/) +- Cloud Native Java +- Cloud Native Python + + +### docker-client 机器 + +- 用于制作docker镜像和测试镜像 + +| name | ip address | location | description | +|----------------------------------|----------------|----------|---------------------------| +| docker-client01v.lobj.juejin.id | 192.168.0.233 | lobj | 本地测试集群01 | + + +### kubernetes cluster list + +| name | location | description | +|---------------------------------------|----------|---------------------------| +| test.kube01.lobj.juejin.id | lobj | 本地测试集群01 | +| beta.kube01.lobj.juejin.id | lobj | 本地beta测试集群01 | +| prod.kube01.qcbj3b.juejin.id | qcbj3b | 青云北京3B区线上集群01 | + + +### ingress 出口设置 + +注意要严格按照列表中的IP和PORT的对应关系来调用PORT, 否则可能会发生ingress流量调度会不起作用或大量流量打到同一个IP上的问题, 业务就无法访问了. + +- test + +| type | ip | port | cluster | instance | +|----------|-----------------|------|------------------------------|----------| +| test | 192.168.0.159 | 80 | test.kube01.lobj.juejin.id | traefik | +| test | 192.168.0.158 | 8000 | test.kube01.lobj.juejin.id | traefik | +| test | 192.168.0.157 | 8080 | test.kube01.lobj.juejin.id | traefik | + +- beta + +| type | ip | port | cluster | instance | +|----------|-----------------|------|------------------------------|----------| +| beta | 192.168.0.99 | 80 | beta.kube01.lobj.juejin.id | traefik | +| beta | 192.168.0.98 | 8000 | beta.kube01.lobj.juejin.id | traefik | +| beta | 192.168.0.97 | 8080 | beta.kube01.lobj.juejin.id | traefik | + +- prod + +| type | ip | port | cluster | instance | comment | +|----------|-----------------|--------|------------------------------|----------|---------------| +| prod | 172.16.0.199 | 80 | prod.kube01.qcbj3b.juejin.id | traefik | | +| prod | 172.16.0.198 | 8000 | prod.kube01.qcbj3b.juejin.id | traefik | | +| prod | 172.16.0.197 | 8080 | prod.kube01.qcbj3b.juejin.id | traefik | | +| prod | 139.198.15.232 | 80/443 | prod.kube01.qcbj3b.juejin.id | traefik | 线上外网出口 | +| prod | 139.198.14.107 | 80/443 | prod.kube01.qcbj3b.juejin.id | traefik | 线上外网出口 | + + + +### test.kube01.lobj.juejin.id 集群设置 + +| ip | hostname | role | disk | +|-----------------|-------------------------------------------|--------------------|--------------| +| 192.168.0.157 | ingress-8080.test.kube01.lobj.juejin.id | ingress-vip | | +| 192.168.0.158 | ingress-8000.test.kube01.lobj.juejin.id | ingress-vip | | +| 192.168.0.159 | ingress-80.test.kube01.lobj.juejin.id | ingress-vip | | +| 192.168.0.160 | test.kube01.lobj.juejin.id | master-vip | | +| 192.168.0.171 | etcd01v.lobj.juejin.id | etcd | 40GB iSCSI | +| 192.168.0.172 | etcd02v.lobj.juejin.id | etcd | 40GB iSCSI | +| 192.168.0.173 | etcd03v.lobj.juejin.id | etcd | 40GB iSCSI | +| 192.168.0.161 | kubernetes-master01v.lobj.juejin.id | kubernetes-master | 100GB iSCSI | +| 192.168.0.162 | kubernetes-master02v.lobj.juejin.id | kubernetes-master | 100GB iSCSI | +| 192.168.0.163 | kubernetes-master03v.lobj.juejin.id | kubernetes-master | 100GB iSCSI | +| 192.168.0.164 | kubernetes-node01v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.165 | kubernetes-node02v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.166 | kubernetes-node03v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.167 | kubernetes-node04v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.168 | kubernetes-node05v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | + + +### beta.kube01.lobj.juejin.id 集群设置 + +| ip | hostname | role | disk | +|-----------------|-------------------------------------------|--------------------|--------------| +| 192.168.0.97 | ingress-8080.beta.kube01.lobj.juejin.id | ingress-vip | | +| 192.168.0.98 | ingress-8000.beta.kube01.lobj.juejin.id | ingress-vip | | +| 192.168.0.99 | ingress-80.beta.kube01.lobj.juejin.id | ingress-vip | | +| 192.168.0.100 | beta.kube01.lobj.juejin.id | master-vip | | +| 192.168.0.121 | etcd04v.lobj.juejin.id | etcd | 40GB iSCSI | +| 192.168.0.122 | etcd05v.lobj.juejin.id | etcd | 40GB iSCSI | +| 192.168.0.123 | etcd06v.lobj.juejin.id | etcd | 40GB iSCSI | +| 192.168.0.101 | kubernetes-master04v.lobj.juejin.id | kubernetes-master | 100GB iSCSI | +| 192.168.0.102 | kubernetes-master05v.lobj.juejin.id | kubernetes-master | 100GB iSCSI | +| 192.168.0.103 | kubernetes-master06v.lobj.juejin.id | kubernetes-master | 100GB iSCSI | +| 192.168.0.104 | kubernetes-node06v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.105 | kubernetes-node07v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.106 | kubernetes-node08v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.107 | kubernetes-node09v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | +| 192.168.0.108 | kubernetes-node10v.lobj.juejin.id | kubernetes-node | 100GB iSCSI | + + +### prod.kube01.qcbj3b.juejin.id 集群设置 + +| ip | hostname | role | disk | +|-----------------|---------------------------------------------|--------------------|------------| +| 172.16.0.197 | ingress-8080.prod.kube01.qcbj3b.juejin.id | ingress-vip | | +| 172.16.0.198 | ingress-8000.prod.kube01.qcbj3b.juejin.id | ingress-vip | | +| 172.16.0.199 | ingress-80.prod.kube01.qcbj3b.juejin.id | ingress-vip | | +| 172.16.0.200 | prod.kube01.qcbj3b.juejin.id | master-vip | | +| 172.16.0.11 | etcd01v.qcbj3b.juejin.id | etcd | 40GB SSD | +| 172.16.0.12 | etcd02v.qcbj3b.juejin.id | etcd | 40GB SSD | +| 172.16.0.13 | etcd03v.qcbj3b.juejin.id | etcd | 40GB SSD | +| 172.16.0.14 | kubernetes-master01v.qcbj3b.juejin.id | kubernetes-master | 100GB SSD | +| 172.16.0.15 | kubernetes-master02v.qcbj3b.juejin.id | kubernetes-master | 100GB SSD | +| 172.16.0.16 | kubernetes-master03v.qcbj3b.juejin.id | kubernetes-master | 100GB SSD | +| 172.16.0.17 | kubernetes-node01v.qcbj3b.juejin.id | kubernetes-node | 100GB SSD | +| 172.16.0.18 | kubernetes-node02v.qcbj3b.juejin.id | kubernetes-node | 100GB SSD | +| 172.16.0.19 | kubernetes-node03v.qcbj3b.juejin.id | kubernetes-node | 100GB SSD | +| 172.16.0.20 | kubernetes-node04v.qcbj3b.juejin.id | kubernetes-node | 100GB SSD | +| 172.16.0.21 | kubernetes-node05v.qcbj3b.juejin.id | kubernetes-node | 100GB SSD | \ No newline at end of file diff --git a/DOCUMENTS/how-to-get-kubectl-bin.md b/DOCUMENTS/how-to-get-kubectl-bin.md new file mode 100644 index 0000000..82ebab0 --- /dev/null +++ b/DOCUMENTS/how-to-get-kubectl-bin.md @@ -0,0 +1,78 @@ +how-to-get-kubectl-bin.md +---------------------- +``` +@version 180902:1 +@author zhangxuhong +``` + +# desc +------ + +如何获取 kubectl 二进制文件. + + +# main +------ + +- 下载 + +根据 kubernetes 文档 [https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) +我们得到 Centos 的 kubectl yum 源在: [https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64](https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64) +我们打开 https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/primary.xml 文件, 按照版本寻找我们要的 kubectl 下载地址. + +``` + +kubectl +x86_64 + + +a554c1728ecf79871b4d3e0fc797568e53149f4ed7ec7e437c949a02f197a1ab + + +Command-line utility for interacting with a Kubernetes cluster. + + +Command-line utility for interacting with a Kubernetes cluster. + + +https://kubernetes.io + +``` + +可以看到 + +``` + +``` + +即为下载地址. 我们拼接url得到下载地址为: +https://packages.cloud.google.com/yum/pool/a554c1728ecf79871b4d3e0fc797568e53149f4ed7ec7e437c949a02f197a1ab-kubectl-1.11.2-0.x86_64.rpm + + +- 解压 rpm 包. + +下载 rpm 包后, 解压即可得到我们需要的 kubectl 二进制文件. +解压需要 rpm2cpio 程序. 这个程序是 rpm 包自带的, 不需要安装. + +``` +wget https://packages.cloud.google.com/yum/pool/a554c1728ecf79871b4d3e0fc797568e53149f4ed7ec7e437c949a02f197a1ab-kubectl-1.11.2-0.x86_64.rpm +rpm2cpio a554c1728ecf79871b4d3e0fc797568e53149f4ed7ec7e437c949a02f197a1ab-kubectl-1.11.2-0.x86_64.rpm | cpio -idmv +``` + +解压后就是了. \ No newline at end of file diff --git a/DOCUMENTS/jenkins-pipline-usage.md b/DOCUMENTS/jenkins-pipline-usage.md new file mode 100644 index 0000000..4b82d33 --- /dev/null +++ b/DOCUMENTS/jenkins-pipline-usage.md @@ -0,0 +1,163 @@ +jenkins-pipline-usage.md +------------------------ +``` +@version 180808:1 +@author zhangxuhong +``` + +Name +---- + + jenkins-pipine-usage - jenkins流水线使用手册. + + +Table of Contents +----------------- + +* [Name](#name) +* [Reference 参考文档](#reference) + + + +Reference 参考文档 +------------------ + +* [官方手册](https://github.com/jenkinsci/kubernetes-plugin) +https://zhangchenchen.github.io/2017/12/17/achieve-cicd-in-kubernetes-with-jenkins/ +https://www.cnblogs.com/hahp/p/5812455.html +https://www.ibm.com/developerworks/cn/devops/d-based-ibm-cloud-private/index.html + + +Tips +---- + +流水线构建过程将每个部分称作"stage", 语法是: +``` +stage('Stage Name') { + // action +} +``` +默认stage中的操作, 例如运行shell命令等, 如果没有指定container, 都是运行是在默认的jnlp容器上的. +如果指定了container, 则会运行在指定的cloud字段中的kubernetes集群中. +所以插件中配置kubernetes集群登录信息完全不是为了直接在上面部署业务, 而是提供了一个运行jenkins slave容器的环境而已. +因此, 传统的 checkout, build, push, deploy 过程中最后的deploy过程就变成了: + - 有几个kubernetes集群就弄几个包含kubectl和认证文件的docker image. + - 部署的时候用这个docker image执行kubectl来部署. + + + + + +下面是一段配置sample: +``` +def label = "worker-${UUID.randomUUID().toString()}" + +podTemplate(label: label, containers: [ + containerTemplate(name: 'gradle', image: 'gradle:4.5.1-jdk9', command: 'cat', ttyEnabled: true), + containerTemplate(name: 'docker', image: 'docker', command: 'cat', ttyEnabled: true), + containerTemplate(name: 'kubectl', image: 'lachlanevenson/k8s-kubectl:v1.8.8', command: 'cat', ttyEnabled: true), + containerTemplate(name: 'helm', image: 'lachlanevenson/k8s-helm:latest', command: 'cat', ttyEnabled: true) +], +volumes: [ + hostPathVolume(mountPath: '/home/gradle/.gradle', hostPath: '/tmp/jenkins/.gradle'), + hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock') +]) { + node(label) { + def myRepo = checkout scm + def gitCommit = myRepo.GIT_COMMIT + def gitBranch = myRepo.GIT_BRANCH + def shortGitCommit = "${gitCommit[0..10]}" + def previousGitCommit = sh(script: "git rev-parse ${gitCommit}~", returnStdout: true) + + stage('Test') { + try { + container('gradle') { + sh """ + pwd + echo "GIT_BRANCH=${gitBranch}" >> /etc/environment + echo "GIT_COMMIT=${gitCommit}" >> /etc/environment + gradle test + """ + } + } + catch (exc) { + println "Failed to test - ${currentBuild.fullDisplayName}" + throw(exc) + } + } + stage('Build') { + container('gradle') { + sh "gradle build" + } + } + stage('Create Docker images') { + container('docker') { + withCredentials([[$class: 'UsernamePasswordMultiBinding', + credentialsId: 'dockerhub', + usernameVariable: 'DOCKER_HUB_USER', + passwordVariable: 'DOCKER_HUB_PASSWORD']]) { + sh """ + docker login -u ${DOCKER_HUB_USER} -p ${DOCKER_HUB_PASSWORD} + docker build -t namespace/my-image:${gitCommit} . + docker push namespace/my-image:${gitCommit} + """ + } + } + } + stage('Run kubectl') { + container('kubectl') { + sh "kubectl get pods" + } + } + stage('Run helm') { + container('helm') { + sh "helm list" + } + } + } +} +``` + + + +Issues +------ + +- 多段构建问题 +``` +Step 1/8 : FROM harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module:latest as template +Error parsing reference: "harbor02.juejin.id/infrastructure/nginx-1.10.3-centos-with-lua-nginx-module:latest as template" is not a valid repository/tag: invalid reference format +``` + +由于线上集群是docker 17.03.2-ce, 不支持多段构建. 所以 xxx as template 语法不能使用. + +- 镜像push问题 +``` ++ docker push harbor02.juejin.id/test/suid-generator:1ea59b9 +The push refers to a repository [harbor02.juejin.id/test/suid-generator] +1c91f1c79983: Preparing +f070d45fa624: Preparing +129f1de793ff: Preparing +0c195f651c3f: Preparing +289b7a478aed: Preparing +ae39cf183283: Preparing +ce860bbdcfdd: Preparing +129b697f70e9: Preparing +ae39cf183283: Waiting +ce860bbdcfdd: Waiting +129b697f70e9: Waiting +denied: requested access to the resource is denied +``` + +线上harbor集群push镜像需要docker登录, 因此需要将使用的构建镜像的docker增加登录信息. 登录信息在/root/.docker/config.json + +- 镜像缓存问题 +containerTemplate中特定tag的image 一旦被下载到本地一次就不会再从harbor下载了, 无论harbor是否更新了image. +所以要使用新的image一定要修改tag. + +- kubeapi版本问题 +``` ++ kubectl --kubeconfig=/root/.kube/config apply -f ./Deployment.yaml +error: unable to recognize "./Deployment.yaml": no matches for kind "Deployment" in version "v1" +``` +Deployment 支持的版本是 apps/v1, 具体可以用kubectl api-versions查看. diff --git a/DOCUMENTS/mapping-external-services.md b/DOCUMENTS/mapping-external-services.md new file mode 100644 index 0000000..6060af1 --- /dev/null +++ b/DOCUMENTS/mapping-external-services.md @@ -0,0 +1,149 @@ +mapping-external-services.md +--------------------------------------------------------- +``` +@version 180807:1 +@author zhangxuhong +``` + +Name +---- + + mapping-external-services - 映射外部服务. + + +Table of Contents +----------------- + +* [Name](#name) +* [Reference 参考文档](#reference) + + + +Reference 参考文档 +------------------ + +* [https://cloudplatform.googleblog.com/2018/05/Kubernetes-best-practices-mapping-external-services.html](https://cloudplatform.googleblog.com/2018/05/Kubernetes-best-practices-mapping-external-services.html) +* [ClusterIP和NodePort的区别](https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0) + +case +---- + +有的时候我们要访问 kubernetes 集群外部的服务. +这些服务可能是没来得及迁移到集群里面的, 或者第三方的, 或者是托管的数据库. +因此便有了映射外部服务的场景. + + +method +------ + +- 映射外部IP和端口 + +- service + +``` +kind: Service +apiVersion: v1 +metadata: + name: mongo + labels: + name: mongo + role: proxy + pl: cpp + application: mongo + version: 16.03 + division: infrastructure +Spec: + type: ClusterIP + ports: + - port: 27017 + targetPort: 27017 +``` + +- endpoints + +``` +kind: Endpoints +apiVersion: v1 +metadata: + name: mongo + labels: + name: mongo + role: proxy + pl: cpp + application: mongo + version: 16.03 + division: infrastructure +subsets: + - addresses: + - ip: 192.168.0.203 + ports: + - port: 27017 +``` + + +- 映射外部URI + +- service + +``` +kind: Service +apiVersion: v1 +metadata: + name: mongo + labels: + name: mongo + role: proxy + pl: cpp + application: mongo + version: 16.03 + division: infrastructure +spec: + type: ExternalName + externalName: mongodb01v.lobj.juejin.id +``` + + +- 映射外部IP并转发端口 + + +- service + +``` +kind: Service +apiVersion: v1 +metadata: + name: mongo + labels: + name: mongo + role: proxy + pl: cpp + application: mongo + version: 16.03 + division: infrastructure +Spec: + type: ClusterIP + ports: + - port: 27017 + targetPort: 7000 +``` + +- endpoints + +``` +kind: Endpoints +apiVersion: v1 +metadata: + name: mongo + labels: + name: mongo + role: proxy + pl: cpp + application: mongo + version: 16.03 + division: infrastructure +subsets: + - addresses: + - ip: 192.168.0.203 + ports: + - port: 7000 +``` \ No newline at end of file diff --git a/README.md b/README.md index 5224ee7..0cafea2 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,30 @@ -# container-guide +README.md +--------- +``` +@version 181008:2 +@author zhangxuhong +``` -container-guide \ No newline at end of file +Name +---- + + container-guide - docker, kubernetes, 容器化教程. + + +Table of Contents +----------------- + +* [Name](#name) +* [Documents](#documents) + + +Documents +--------- + +- [container guide (容器化参考文档)](./DOCUMENTS/container-guide.md) +- [Build A Docker Image (构建Docker镜像)](./DOCUMENTS/build-a-docker-image.md) +- [Configuring Certificates For Jenkins Kubernetes Plugin (配置jenkins kubernetes插件的认证)](./DOCUMENTS/configuring-certificates-for-jenkins-kubernetes-plugin.md) +- [Jenkins Pipine Usage (jenkins流水线使用手册)](./DOCUMENTS/jenkins-pipline-usage.md) +- [how to get kubectl bin (如何获取kubectl二进制文件)](./DOCUMENTS/how-to-get-kubectl-bin.md) +- [jenkins pipline usage (jenkins流水线使用手册)](./DOCUMENTS/jenkins-pipline-usage.md) +- [mapping external services (kubernetes映射外部服务)](./DOCUMENTS/mapping-external-services.md)