0. 起因

在我们测试环境k8s上运行了数十个helm release, 昨天helm ls的时候居然报错了

$ helm ls Error: trying to send message larger than max (23934705 vs. 20971520)

这个问题搜索下, 发现github issue今年一月就已经有人遇到了 https://github.com/helm/helm/issues/3322 , 今年二月份官方的同学也提pr修复了 https://github.com/helm/helm/pull/3514

为啥我这里还会出现呢

1. 探究

看下修复pr的主要内容, 发现只是把限制从4M提升到20M...

这个值还是一个常量, 目前最近的release和master也一样 我们使用的tiller镜像 registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1，已经包了修复内容,看来想解决只能自己修改源码编译了

2. 修复

2.1 修改源码

const maxMsgSize = 1024 * 1024 * 100
这里改成100M,还能超的话只能说要再来一个集群了..

2.2 编译 Helm

拉下代码, 发现helm使用make进行构建, 按照官方 https://docs.helm.sh/developers/, 尝试初始化和构建

17:03 $ make bootstrap build glide install --strip-vendor [INFO] Downloading dependencies. Please wait... [INFO] --> Found desired version locally github.com/aokoli/goutils 9c37978a95bd5c709a15883b6242714ea6709e64! [INFO] --> Found desired version locally github.com/asaskevich/govalidator 7664702784775e51966f0885f5cd27435916517b! [INFO] --> Found desired version locally github.com/Azure/go-ansiterm d6e3b3328b783f23731bc4d058875b0371ff8109! [INFO] --> Found desired version locally github.com/Azure/go-autorest bca49d5b51a50dc5bb17bbf6204c711c6dbded06! [INFO] --> Found desired version locally github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4! [INFO] --> Found desired version locally github.com/BurntSushi/toml 3012a1dbe2e4bd1391d42b32f0577cb7bbc7f005! [INFO] --> Found desired version locally github.com/chai2010/gettext-go c6fed771bfd517099caf0f7a961671fa8ed08723! [INFO] --> Found desired version locally github.com/cpuguy83/go-md2man 71acacd42f85e5e82f70a55327789582a5200a90! [INFO] --> Found desired version locally github.com/cyphar/filepath-securejoin a261ee33d7a517f054effbf451841abaafe3e0fd! [INFO] --> Found desired version locally github.com/davecgh/go-spew 782f4967f2dc4564575ca782fe2d04090b5faca8! [INFO] --> Found desired version locally github.com/dgrijalva/jwt-go 01aeca54ebda6e0fbfafd0a524d234159c05ec20! [INFO] --> Found desired version locally github.com/docker/distribution edc3ab29cdff8694dd6feb85cfeb4b5f1b38ed9c! [INFO] --> Found desired version locally github.com/docker/docker a9fbbdc8dd8794b20af358382ab780559bca589d! [INFO] --> Found desired version locally github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d! [INFO] --> Found desired version locally github.com/docker/go-units 9e638d38cf6977a37a8ea0078f3ee75a7cdb2dd1! [INFO] --> Found desired version locally github.com/docker/spdystream 449fdfce4d962303d702fec724ef0ad181c92528! [INFO] --> Found desired version locally github.com/evanphx/json-patch 36442dbdb585210f8d5a1b45e67aa323c197d5c4! [INFO] --> Found desired version locally github.com/exponent-io/jsonpath d6023ce2651d8eafb5c75bb0c7167536102ec9f5! [INFO] --> Found desired version locally github.com/fatih/camelcase f6a740d52f961c60348ebb109adde9f4635d7540! [INFO] --> Found desired version locally github.com/ghodss/yaml 73d445a93680fa1a78ae23a5839bad48f32ba1ee! [INFO] --> Found desired version locally github.com/go-openapi/jsonpointer 46af16f9f7b149af66e5d1bd010e3574dc06de98! [INFO] --> Found desired version locally github.com/go-openapi/jsonreference 13c6e3589ad90f49bd3e3bbe2c2cb3d7a4142272! [INFO] --> Found desired version locally github.com/go-openapi/spec 1de3e0542de65ad8d75452a595886fdd0befb363! [INFO] --> Found desired version locally github.com/go-openapi/swag f3f9494671f93fcff853e3c6e9e948b3eb71e590! [INFO] --> Found desired version locally github.com/gobwas/glob 5ccd90ef52e1e632236f7326478d4faa74f99438! [INFO] --> Found desired version locally github.com/gogo/protobuf c0656edd0d9eab7c66d1eb0c568f9039345796f7! [INFO] --> Found desired version locally github.com/golang/glog 44145f04b68cf362d9c4df2182967c2275eaefed! [INFO] --> Found desired version locally github.com/golang/groupcache 02826c3e79038b59d737d3b1c0a1d937f71a4433! [INFO] --> Found desired version locally github.com/golang/protobuf 1643683e1b54a9e88ad26d98f81400c8c9d9f4f9! [INFO] --> Found desired version locally github.com/google/btree 7d79101e329e5a3adf994758c578dab82b90c017! [INFO] --> Found desired version locally github.com/google/gofuzz 44d81051d367757e1c7c6a5a86423ece9afcf63c! [INFO] --> Found desired version locally github.com/google/uuid 064e2069ce9c359c118179501254f67d7d37ba24! [INFO] --> Found desired version locally github.com/googleapis/gnostic 0c5108395e2debce0d731cf0287ddf7242066aba! [INFO] --> Found desired version locally github.com/gophercloud/gophercloud 781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d! [INFO] --> Found desired version locally github.com/gosuri/uitable 36ee7e946282a3fb1cfecd476ddc9b35d8847e42! [INFO] --> Found desired version locally github.com/gregjones/httpcache 787624de3eb7bd915c329cba748687a3b22666a6! [INFO] --> Found desired version locally github.com/grpc-ecosystem/go-grpc-prometheus 0c1b191dbfe51efdabe3c14b9f6f3b96429e0722! [INFO] --> Found desired version locally github.com/hashicorp/golang-lru a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4! [INFO] --> Found desired version locally github.com/huandu/xstrings 3959339b333561bf62a38b424fd41517c2c90f40! [INFO] --> Found desired version locally github.com/imdario/mergo 9316a62528ac99aaecb4e47eadd6dc8aa6533d58! [INFO] --> Found desired version locally github.com/inconshreveable/mousetrap 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75! [INFO] --> Found desired version locally github.com/json-iterator/go f2b4162afba35581b6d4a50d3b8f34e33c144682! [INFO] --> Found desired version locally github.com/mailru/easyjson 2f5df55504ebc322e4d52d34df6a1f5b503bf26d! [INFO] --> Found desired version locally github.com/MakeNowJust/heredoc bb23615498cded5e105af4ce27de75b089cbe851! [INFO] --> Found desired version locally github.com/Masterminds/semver 517734cc7d6470c0d07130e40fd40bdeb9bcd3fd! [INFO] --> Found desired version locally github.com/Masterminds/sprig 15f9564e7e9cf0da02a48e0d25f12a7b83559aa6! [WARN] Unable to checkout golang.org/x/text [ERROR] Update failed for golang.org/x/text: Cannot detect VCS [WARN] Unable to checkout golang.org/x/time [ERROR] Update failed for golang.org/x/time: Cannot detect VCS [WARN] Unable to checkout google.golang.org/appengine [ERROR] Update failed for google.golang.org/appengine: Cannot detect VCS [WARN] Unable to checkout google.golang.org/genproto [ERROR] Update failed for google.golang.org/genproto: Cannot detect VCS [WARN] Unable to checkout google.golang.org/grpc [ERROR] Update failed for google.golang.org/grpc: Cannot detect VCS [ERROR] Failed to install: Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS Cannot detect VCS make: *** [bootstrap] Error 1

问题1: 国内golang的依赖包各种被墙

.PHONY: bootstrap bootstrap: ifndef HAS_GLIDE go get -u github.com/Masterminds/glide endif ifndef HAS_GOX go get -u github.com/mitchellh/gox endif ifndef HAS_GIT $(error You must install Git) endif glide install --strip-vendor go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go

看一眼Makefile, 这里用的glide管理的依赖包

好在github基本都会有对应的包源码, 有几种解决方式:

1. clone源码然后按照目录结构放到对应$GOPATH/src下
2. 给git设置proxy,小飞机你懂的~
3. 使用 gopm
4. 使用glide的mirror功能设置映射 glide-mirror

这里用的方法4, mirrors.yaml, 不想一个一个做映射可以参考这里 mirrors.yaml

问题2: 编译路径

17:36 $ make build GOBIN=/home/jokimina/golang/k8s.io/helm/bin go install -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=26c88ec815785708cc19c014366ce4bb000ae89c -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/... warning: "k8s.io/helm/cmd/..." matched no packages

如果是直接拉下代码放到随意的目录, build的时候会报错, 需要放到
$GOPATH/src/k8s.io/helm再执行编译

问题3: golang默认不是静态编译, docker alpine镜像没有运行所需要的库

$ make build GOBIN=/home/xxx/go/src/k8s.io/helm/bin CGO_ENABLED=0 go install -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=147c8217c7843b53e98528f6df12890d64c696be -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/... ~/go/src/k8s.io/helm $ ls bin/ helm rudder tiller

make bootstrap build执行完后会生成二进制文件到 $GOBIN 目录下 , 直接替换掉阿里云tiller容器内的运行发现报错

# /tiller sh: /tiller: not found

但是本地可以运行, 搜索下 https://stackoverflow.com/questions/36279253/go-compiled-binary-wont-run-in-an-alpine-docker-container-on-ubuntu-host

发现是由于默认没有静态编译导致引用了alpine不存在的库链接

# ldd tiller /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000) libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000) libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)

重新静态编译下, 看到Makefile默认支持交叉编译这里偷个懒

make TARGETS="linux/amd64" build-cross

节省时间这里就只编译需要的了, 构建好的二进制文件会在 _dist/下

2.3 构建镜像

这里就直接替换tiller然后commit镜像, 不写Dockerfile了

$ docker pull registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1 $ docker run -d --name build registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1 $ docker cp tiller build:/tiller $ docker commit build registry-vpc.cn-beijing.aliyuncs.com/xxx-service/helm:v2.11.0 $ docker push !$

2.4 换掉k8s集群上的tiller

$ helm init --tiller-image registry-vpc.cn-beijing.aliyuncs.com/xxx-service/helm:v2.11.0 --upgrade $ helm version Client: &version.Version{SemVer:"v2.9.1", GitCommit:"08c1144f5eb3e3b636d9775617287cc26e53dba4", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.9.1", GitCommit:"e8c5cf08751dfc7be5c4bedcd400ed78355e3468", GitTreeState:"dirty"} 

测试下

ok了

3. 后话

• 在配置参数不变更只是更新镜像的情况下, 其实可以考虑直接update deplayment开销会小很多, helm主要还是首次初始化的时候自动生成了需要的resource.
• 正常情况下官方限制的20M基本能满足需求, 只是超了一点点的话可以在helm ls的时候用-m或者--deployed限制下返回数量和过滤状态
  不

过这里是一个测试集群release会不断增加...所以无法忍受了