esrally测试工具，针对现有集群，自定义track
安装 Python 3.8+ including pip3, git 1.9+ pip3 install esrally 高级配置 esrally configure --advanced-config （可多次执行） Enter the benchmark root directory (default: /home/apps/.rally/benchmarks): Using default value '/home/apps/.rally/benchmarks' Enter your Elasticsearch project directory: (default: /home/apps/.rally/benchmarks/src/elasticsearch): Using default value '/home/apps/.rally/benchmarks/src/elasticsearch' Where should metrics be kept? (1) In memory (simpler but less options for analysis) (2) Elasticsearch (requires a separate ES instance, keeps all raw samples for analysis) (default: 1): 2 Enter the host name of the ES metrics store (default: localhost): Using default value 'localhost' Enter the port of the ES metrics store: 9200 Use secure connection (True, False) (default: False): Username for basic authentication (empty if not needed) (default: ): Password for basic authentication (empty if not needed) (default: ): Enter a descriptive name for this benchmark environment (ASCII, no spaces) (default: local): rally_demo Do you want Rally to keep the Elasticsearch benchmark candidate installation including the index (will use several GB per trial run)? (default: False): Configuration successfully written to /home/apps/.rally/rally.ini. Happy benchmarking! rally-metrics-*：记录race中所有的采样数据（性能报告数据都是基于采样数据汇总得出的结果），可以通过采样数据看到具体抖动情况进行具体分析 rally-results-*：基于上面的采样数据进行详细的数据汇总，针对每个指标进行汇总，可以针对每个指标进行最终的结果展示 rally-races-：把最终性能压测结果（即我们看到的输出数据）展示出来，一个文档就是一次race压测 将测试结果放入开启x_pack集群中需要配置 vim .rally/rally.ini [reporting] #datastore.ssl.verification_mode = none datastore.ssl.certificate_authorities = /home/elasticsearch/client-ca.cer 使用 配置代理下载track export http_proxy=http://proxy.acme.org:8888/ /root/.rally/logs/rally.log将记录 Rally connects via proxy URL [http://proxy.acme.org:3128/] to the Internet (picked up from the environment variable [http_proxy]). esrally list tracks 类似于赛道的意思 nested 嵌套文档 percolator 过滤查询 eventdata geoshape geopoint 地理查询 so geopointshape noaa metricbeat nyc_taxis 高度数据化结构 http_logs pmc 全文搜索 geonames 结构化数据 对现有集群进行基准测试 esrally --track=pmc --target-hosts=10.5.5.10:9243,10.5.5.11:9243,10.5.5.12:9243 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'changeme'" --user-tag="intention:baseline_github_1234" --report-file=result.csv --report-format=csv rally可以组成分布式集群，并可以根据配置文件进行测试，因为没有试过，这里不再赘述 esrally list races 对比两场比赛 esrally compare --baseline=0bfd4542-3821-4c79-81a2-0858636068ce --contender=beb154e4-0a05-4f45-ad9f-e34f9a9e51f7 创建track mkdir .rally/benchmarks/data/XXX 获取数据 allCountries.txt 转换为json格式 vim toJSON.py import json cols = (("geonameid", "int", True), ("name", "string", True), ("asciiname", "string", False), ("alternatenames", "string", False), ("latitude", "double", True), ("longitude", "double", True), ("feature_class", "string", False), ("feature_code", "string", False), ("country_code", "string", True), ("cc2", "string", False), ("admin1_code", "string", False), ("admin2_code", "string", False), ("admin3_code", "string", False), ("admin4_code", "string", False), ("population", "long", True), ("elevation", "int", False), ("dem", "string", False), ("timezone", "string", False)) def main(): with open("allCountries.txt", "rt", encoding="UTF-8") as f: for line in f: tup = line.strip().split("\t") record = {} for i in range(len(cols)): name, type, include = cols[i] if tup[i] != "" and include: if type in ("int", "long"): record[name] = int(tup[i]) elif type == "double": record[name] = float(tup[i]) elif type == "string": record[name] = tup[i] print(json.dumps(record, ensure_ascii=False)) if __name__ == "__main__": main() python3 toJSON.py > documents.json 编写track.json 检查是否可用esrally info --track-path=~/rally-tracks/tutorial track解剖 {% set index_count = 2 %} #设置循环次数 { "version": 2, "description": "Tutorial benchmark for Rally", "indices": [ { "name": "geonames", #要操作索引的名称 "body": "index.json" } ], "corpora": [ #此轨道使用的所有文档语料库 { "name": "rally-tutorial", "documents": [ { "source-file": "documents.json", "document-count": 11658903, "uncompressed-bytes": 1544799789 } ] } ], "schedule": [ #具体执行的操作 { "operation": { "operation-type": "delete-index" #操作类型 } }, { "operation": { "operation-type": "create-index" } }, { "operation": { "operation-type": "cluster-health", "request-params": { "wait_for_status": "green" } } }, { "operation": { "operation-type": "bulk", "bulk-size": 5000 }, "warmup-time-period": 120, "clients": 8 }, { "operation": { "operation-type": "force-merge" } }, { "operation": { "name": "query-match-all", "operation-type": "search", "body": { "query": { "match_all": {} } } }, "clients": 8, #发起请求客户端数量 "warmup-iterations": 1000, #预热，不显示在测量结果中 "iterations": 1000, #执行次数 "target-throughput": 100 #尽可能慢地运行任务，target-throughput/clients/s qps } ] } 配置parallel实现并行执行语句 https://esrally.readthedocs.io/en/latest/track.html https://elasticsearch-benchmarks.elastic.co/# https://learnku.com/articles/41047
