【Helm】release太多导致报错,一次从问题排查,修改源码编译到构建tiller镜像修复的经历

2018-10-12 725

0. 起因

在我们测试环境k8s上运行了数十个helm release, 昨天helm ls的时候居然报错了

$ helm ls
Error: trying to send message larger than max (23934705 vs. 20971520)

这个问题搜索下, 发现github issue今年一月就已经有人遇到了 https://github.com/helm/helm/issues/3322 , 今年二月份官方的同学也提pr修复了 https://github.com/helm/helm/pull/3514

为啥我这里还会出现呢

1. 探究

看下修复pr的主要内容, 发现只是把限制从4M提升到20M...

这个值还是一个常量, 目前最近的release和master也一样我们使用的tiller镜像 registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1，已经包了修复内容,看来想解决只能自己修改源码编译了

2. 修复

2.1 修改源码

const maxMsgSize = 1024 * 1024 * 100
这里改成100M,还能超的话只能说要再来一个集群了..

2.2 编译 Helm

拉下代码, 发现helm使用make进行构建, 按照官方 https://docs.helm.sh/developers/, 尝试初始化和构建

17:03 $ make bootstrap build
glide install --strip-vendor
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Found desired version locally github.com/aokoli/goutils 9c37978a95bd5c709a15883b6242714ea6709e64!
[INFO]  --> Found desired version locally github.com/asaskevich/govalidator 7664702784775e51966f0885f5cd27435916517b!
[INFO]  --> Found desired version locally github.com/Azure/go-ansiterm d6e3b3328b783f23731bc4d058875b0371ff8109!
[INFO]  --> Found desired version locally github.com/Azure/go-autorest bca49d5b51a50dc5bb17bbf6204c711c6dbded06!
[INFO]  --> Found desired version locally github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4!
[INFO]  --> Found desired version locally github.com/BurntSushi/toml 3012a1dbe2e4bd1391d42b32f0577cb7bbc7f005!
[INFO]  --> Found desired version locally github.com/chai2010/gettext-go c6fed771bfd517099caf0f7a961671fa8ed08723!
[INFO]  --> Found desired version locally github.com/cpuguy83/go-md2man 71acacd42f85e5e82f70a55327789582a5200a90!
[INFO]  --> Found desired version locally github.com/cyphar/filepath-securejoin a261ee33d7a517f054effbf451841abaafe3e0fd!
[INFO]  --> Found desired version locally github.com/davecgh/go-spew 782f4967f2dc4564575ca782fe2d04090b5faca8!
[INFO]  --> Found desired version locally github.com/dgrijalva/jwt-go 01aeca54ebda6e0fbfafd0a524d234159c05ec20!
[INFO]  --> Found desired version locally github.com/docker/distribution edc3ab29cdff8694dd6feb85cfeb4b5f1b38ed9c!
[INFO]  --> Found desired version locally github.com/docker/docker a9fbbdc8dd8794b20af358382ab780559bca589d!
[INFO]  --> Found desired version locally github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d!
[INFO]  --> Found desired version locally github.com/docker/go-units 9e638d38cf6977a37a8ea0078f3ee75a7cdb2dd1!
[INFO]  --> Found desired version locally github.com/docker/spdystream 449fdfce4d962303d702fec724ef0ad181c92528!
[INFO]  --> Found desired version locally github.com/evanphx/json-patch 36442dbdb585210f8d5a1b45e67aa323c197d5c4!
[INFO]  --> Found desired version locally github.com/exponent-io/jsonpath d6023ce2651d8eafb5c75bb0c7167536102ec9f5!
[INFO]  --> Found desired version locally github.com/fatih/camelcase f6a740d52f961c60348ebb109adde9f4635d7540!
[INFO]  --> Found desired version locally github.com/ghodss/yaml 73d445a93680fa1a78ae23a5839bad48f32ba1ee!
[INFO]  --> Found desired version locally github.com/go-openapi/jsonpointer 46af16f9f7b149af66e5d1bd010e3574dc06de98!
[INFO]  --> Found desired version locally github.com/go-openapi/jsonreference 13c6e3589ad90f49bd3e3bbe2c2cb3d7a4142272!
[INFO]  --> Found desired version locally github.com/go-openapi/spec 1de3e0542de65ad8d75452a595886fdd0befb363!
[INFO]  --> Found desired version locally github.com/go-openapi/swag f3f9494671f93fcff853e3c6e9e948b3eb71e590!
[INFO]  --> Found desired version locally github.com/gobwas/glob 5ccd90ef52e1e632236f7326478d4faa74f99438!
[INFO]  --> Found desired version locally github.com/gogo/protobuf c0656edd0d9eab7c66d1eb0c568f9039345796f7!
[INFO]  --> Found desired version locally github.com/golang/glog 44145f04b68cf362d9c4df2182967c2275eaefed!
[INFO]  --> Found desired version locally github.com/golang/groupcache 02826c3e79038b59d737d3b1c0a1d937f71a4433!
[INFO]  --> Found desired version locally github.com/golang/protobuf 1643683e1b54a9e88ad26d98f81400c8c9d9f4f9!
[INFO]  --> Found desired version locally github.com/google/btree 7d79101e329e5a3adf994758c578dab82b90c017!
[INFO]  --> Found desired version locally github.com/google/gofuzz 44d81051d367757e1c7c6a5a86423ece9afcf63c!
[INFO]  --> Found desired version locally github.com/google/uuid 064e2069ce9c359c118179501254f67d7d37ba24!
[INFO]  --> Found desired version locally github.com/googleapis/gnostic 0c5108395e2debce0d731cf0287ddf7242066aba!
[INFO]  --> Found desired version locally github.com/gophercloud/gophercloud 781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d!
[INFO]  --> Found desired version locally github.com/gosuri/uitable 36ee7e946282a3fb1cfecd476ddc9b35d8847e42!
[INFO]  --> Found desired version locally github.com/gregjones/httpcache 787624de3eb7bd915c329cba748687a3b22666a6!
[INFO]  --> Found desired version locally github.com/grpc-ecosystem/go-grpc-prometheus 0c1b191dbfe51efdabe3c14b9f6f3b96429e0722!
[INFO]  --> Found desired version locally github.com/hashicorp/golang-lru a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4!
[INFO]  --> Found desired version locally github.com/huandu/xstrings 3959339b333561bf62a38b424fd41517c2c90f40!
[INFO]  --> Found desired version locally github.com/imdario/mergo 9316a62528ac99aaecb4e47eadd6dc8aa6533d58!
[INFO]  --> Found desired version locally github.com/inconshreveable/mousetrap 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75!
[INFO]  --> Found desired version locally github.com/json-iterator/go f2b4162afba35581b6d4a50d3b8f34e33c144682!
[INFO]  --> Found desired version locally github.com/mailru/easyjson 2f5df55504ebc322e4d52d34df6a1f5b503bf26d!
[INFO]  --> Found desired version locally github.com/MakeNowJust/heredoc bb23615498cded5e105af4ce27de75b089cbe851!
[INFO]  --> Found desired version locally github.com/Masterminds/semver 517734cc7d6470c0d07130e40fd40bdeb9bcd3fd!
[INFO]  --> Found desired version locally github.com/Masterminds/sprig 15f9564e7e9cf0da02a48e0d25f12a7b83559aa6!
[WARN]  Unable to checkout golang.org/x/text
[ERROR] Update failed for golang.org/x/text: Cannot detect VCS
[WARN]  Unable to checkout golang.org/x/time
[ERROR] Update failed for golang.org/x/time: Cannot detect VCS
[WARN]  Unable to checkout google.golang.org/appengine
[ERROR] Update failed for google.golang.org/appengine: Cannot detect VCS
[WARN]  Unable to checkout google.golang.org/genproto
[ERROR] Update failed for google.golang.org/genproto: Cannot detect VCS
[WARN]  Unable to checkout google.golang.org/grpc
[ERROR] Update failed for google.golang.org/grpc: Cannot detect VCS
[ERROR] Failed to install: Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
Cannot detect VCS
make: *** [bootstrap] Error 1

问题1: 国内golang的依赖包各种被墙

.PHONY: bootstrap
bootstrap:
ifndef HAS_GLIDE
        go get -u github.com/Masterminds/glide
endif
ifndef HAS_GOX
        go get -u github.com/mitchellh/gox
endif

ifndef HAS_GIT
        $(error You must install Git)
endif
        glide install --strip-vendor
        go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go

看一眼Makefile, 这里用的glide管理的依赖包

好在github基本都会有对应的包源码, 有几种解决方式:

clone源码然后按照目录结构放到对应$GOPATH/src下
给git设置proxy,小飞机你懂的~
使用 gopm
使用glide的mirror功能设置映射 glide-mirror

这里用的方法4, mirrors.yaml, 不想一个一个做映射可以参考这里 mirrors.yaml

问题2: 编译路径

17:36 $ make build
GOBIN=/home/jokimina/golang/k8s.io/helm/bin go install  -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=26c88ec815785708cc19c014366ce4bb000ae89c -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/...
warning: "k8s.io/helm/cmd/..." matched no packages

如果是直接拉下代码放到随意的目录, build的时候会报错, 需要放到
$GOPATH/src/k8s.io/helm再执行编译

问题3: golang默认不是静态编译, docker alpine镜像没有运行所需要的库

$ make build
GOBIN=/home/xxx/go/src/k8s.io/helm/bin CGO_ENABLED=0 go install  -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=147c8217c7843b53e98528f6df12890d64c696be -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/...
 ~/go/src/k8s.io/helm 
$ ls bin/
helm  rudder  tiller

make bootstrap build执行完后会生成二进制文件到 $GOBIN 目录下 , 直接替换掉阿里云tiller容器内的运行发现报错

# /tiller 
sh: /tiller: not found

但是本地可以运行, 搜索下 https://stackoverflow.com/questions/36279253/go-compiled-binary-wont-run-in-an-alpine-docker-container-on-ubuntu-host

发现是由于默认没有静态编译导致引用了alpine不存在的库链接

# ldd tiller 
    /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)

重新静态编译下, 看到Makefile默认支持交叉编译这里偷个懒

make TARGETS="linux/amd64" build-cross

节省时间这里就只编译需要的了, 构建好的二进制文件会在 _dist/下

2.3 构建镜像

这里就直接替换tiller然后commit镜像, 不写Dockerfile了

$ docker pull registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1
$ docker run -d --name build registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1
$ docker cp tiller build:/tiller
$ docker commit build registry-vpc.cn-beijing.aliyuncs.com/xxx-service/helm:v2.11.0
$ docker push !$

2.4 换掉k8s集群上的tiller

$ helm init --tiller-image registry-vpc.cn-beijing.aliyuncs.com/xxx-service/helm:v2.11.0 --upgrade
$ helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"08c1144f5eb3e3b636d9775617287cc26e53dba4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"e8c5cf08751dfc7be5c4bedcd400ed78355e3468", GitTreeState:"dirty"}

测试下

ok了

3. 后话

在配置参数不变更只是更新镜像的情况下, 其实可以考虑直接update deplayment开销会小很多, helm主要还是首次初始化的时候自动生成了需要的resource.
正常情况下官方限制的20M基本能满足需求, 只是超了一点点的话可以在helm ls的时候用-m或者--deployed限制下返回数量和过滤状态
不

过这里是一个测试集群release会不断增加...所以无法忍受了

微信关注我们

原文链接：https://yq.aliyun.com/articles/652395

转载内容版权归作者及来源网站所有！

低调大师中文资讯倾力打造互联网数据资讯、行业资源、电子商务、移动互联网、网络营销平台。持续更新报道IT业界、互联网、市场资讯、驱动更新,是最及时权威的产业资讯及硬件资讯报道平台。

云服务ecs与云虚拟主机应该如何选择？

随着网络时代的发展，越来越多的网站和个人都有企业网站、个人网站的需求。而我们部署网站时一般有两种选择：云服务器、云虚拟主机，那么这两者之间有何区别呢？我们应该如何根据自己的实际情况选择呢？ 1.方便程度比较(云虚拟主机胜) 云虚拟主机：代码运行环境（php|.net）数据库（mysql|sqlserver）在购买完成后就已经基本配置完成了，只需要根据自己的代码调整相关版本设置即可发布访问，网站的开启关闭也只需要在控制中心进行操作 ecs服务器：购买完成后相当于是一台刚安装完系统（window|linux）的新电脑，用户需要自己登录主机安装相应的代码运行环境和数据库，并完成相关的配置，这对很多新手来说需要学习成本 2.灵活性比较（ecs服务器胜）云虚拟主机：购买之后，代码运行环境和数据库类型就已经确定，用户只能调整相关的版本，不能修改环境（php-》.net）,数据库也是一样 ecs服务器：购买后用户可根据自己的情况随时安装或卸载需要的环境及数据库，完全由自己控制，甚至可以重转操作系统（windows-》Linux） 3.配置比较（ecs服务器胜）云虚...

2018-10-13

667

ECS 云盘日常操作：购买并挂载新云盘到 ECS 服务器、云盘扩容、云盘设备名的注意事项、开机自动挂载云盘的注意点；注意，这里所讲的【不是】【随实例一起创建的云盘】，而是，事后需要增加 ECS 的数据盘新购或扩容；另外，ECS 也可以选择挂载 NAS，这个要根据具体的业务场景来选择；我们的选择标准是：NAS 可以挂载在多台 ECS 上，通过 NFS 协议实现磁盘的读写；而云盘只能挂载在一台 ECS 上，相当于 ECS 新接了一个外接硬盘；如果业务需要多台 ECS 同时使用一块存储的话，可以选择 NAS；价格方面，SSD 类型的存储 NAS 要比云盘贵 50%多一点，高效云盘类型 NAS 要比云盘贵15%左右；这里讲的是【云盘】而不是【NAS】的操作；一、新云盘购买、挂载参考文献A【购买】：https://help.aliyun.com/document_detail/25445.html?spm=a2c4g.11186623.6.679.5e561846LYJYXV 参考文献B【挂载】：https://help.aliyun.com/document_detail/2...

2018-10-14

889

资源下载

更多资源

优质分享App

近一个月的开发和优化，本站点的第一个app全新上线。该app采用极致压缩，本体才4.36MB。系统里面做了大量数据访问、缓存优化。方便用户在手机上查看文章。后续会推出HarmonyOS的适配版本。

腾讯云软件源

为解决软件依赖安装时官方源访问速度慢的问题，腾讯云为一些软件搭建了缓存服务。您可以通过使用腾讯云软件源站来提升依赖包的安装速度。为了方便用户自由搭建服务架构，目前腾讯云软件源站支持公网访问和内网访问。

Nacos

Nacos /nɑ:kəʊs/ 是 Dynamic Naming and Configuration Service 的首字母简称，一个易于构建 AI Agent 应用的动态服务发现、配置管理和AI智能体管理平台。Nacos 致力于帮助您发现、配置和管理微服务及AI智能体应用。Nacos 提供了一组简单易用的特性集，帮助您快速实现动态服务发现、服务配置、服务元数据、流量管理。Nacos 帮助您更敏捷和容易地构建、交付和管理微服务平台。

Rocky Linux

Rocky Linux（中文名：洛基）是由Gregory Kurtzer于2020年12月发起的企业级Linux发行版，作为CentOS稳定版停止维护后与RHEL（Red Hat Enterprise Linux）完全兼容的开源替代方案，由社区拥有并管理，支持x86_64、aarch64等架构。其通过重新编译RHEL源代码提供长期稳定性，采用模块化包装和SELinux安全架构，默认包含GNOME桌面环境及XFS文件系统，支持十年生命周期更新。